Sep 10 13:30:22.705324 ip-10-0-133-208 systemd[1]: Starting Kubernetes Kubelet... Sep 10 13:30:23.124423 ip-10-0-133-208 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:30:23.124423 ip-10-0-133-208 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 13:30:23.124423 ip-10-0-133-208 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:30:23.124423 ip-10-0-133-208 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 13:30:23.124423 ip-10-0-133-208 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 13:30:23.124981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.124453 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 13:30:23.128467 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128452 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128468 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128473 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128477 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128480 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128483 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128486 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128489 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128492 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128495 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128498 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128501 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128504 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128507 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.128505 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128510 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128515 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128518 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128521 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128524 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128527 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128533 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128536 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128539 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128541 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128544 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128546 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128549 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128551 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128555 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128557 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128559 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128562 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128564 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128567 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.128847 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128570 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128572 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128575 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128578 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128582 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128585 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128587 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128590 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128592 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128594 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128597 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128600 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128602 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128605 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128609 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128612 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128614 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128617 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128619 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.129378 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128622 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128625 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128627 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128630 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128632 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128635 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128637 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128640 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128642 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128645 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128648 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128650 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128653 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128657 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128659 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128662 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128664 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128667 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128669 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128672 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.129840 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128674 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128677 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128679 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128688 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128692 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128695 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128697 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128702 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128705 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128708 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128710 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128713 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128716 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128718 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128721 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128723 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128726 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128728 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128731 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128734 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.130329 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128736 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128738 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128741 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128744 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128746 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.128749 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129449 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129456 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129459 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129461 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129464 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129467 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129470 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129473 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129475 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129480 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129483 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129486 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.130805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129490 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129494 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129497 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129500 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129503 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129506 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129510 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129513 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129516 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129519 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129522 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129525 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129528 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129531 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129534 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129536 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129539 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129542 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129544 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129547 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.131258 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129550 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129552 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129555 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129557 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129560 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129563 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129565 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129568 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129570 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129573 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129575 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129578 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129581 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129584 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129586 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129589 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129593 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129596 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129598 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.131785 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129601 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129603 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129606 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129608 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129611 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129613 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129616 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129618 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129620 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129623 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129625 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129628 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129630 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129633 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129635 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129639 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129641 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129644 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129646 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.132253 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129649 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129651 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129654 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129657 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129659 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129662 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129665 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129667 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129670 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129672 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129676 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129678 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129681 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129683 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129686 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129688 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129690 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129693 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129695 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129698 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129700 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.132725 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129703 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129705 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129708 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129710 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129713 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129717 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129720 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.129724 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130104 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130109 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130113 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130116 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130119 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130123 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130126 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130129 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130132 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130134 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130138 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.133248 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130140 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130143 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130146 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130149 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130152 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130155 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130157 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130160 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130164 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130168 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130170 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130173 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130175 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130178 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130180 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130187 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130190 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130193 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130196 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.133700 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130199 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130202 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130204 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130207 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130210 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130212 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130215 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130217 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130220 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130223 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130227 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130232 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130235 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130237 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130240 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130243 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130247 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130249 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130253 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130255 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.134178 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130258 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130261 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130263 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130265 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130268 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130271 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130273 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130276 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130279 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130282 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130285 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130287 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130290 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130292 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130295 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130297 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130300 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130302 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130305 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130307 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.134649 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130310 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130312 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130315 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130319 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130322 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130325 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130327 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130330 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130332 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130335 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130338 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130341 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130344 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130346 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130349 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130351 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130354 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130357 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130361 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130363 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.135166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130368 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130370 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130373 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130375 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130378 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130380 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130383 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130386 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.130388 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130459 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130506 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130512 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130516 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130520 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130523 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130528 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130532 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130535 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130538 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130542 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130545 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130548 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 13:30:23.135639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130552 2559 flags.go:64] FLAG: --cgroup-root="" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130555 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130558 2559 flags.go:64] FLAG: --client-ca-file="" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130561 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130564 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130569 2559 flags.go:64] FLAG: --cluster-domain="" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130571 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130574 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130577 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130581 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130586 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130589 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130592 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130595 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130598 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130601 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130604 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130607 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130610 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130614 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130617 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130620 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130623 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130626 2559 flags.go:64] FLAG: --enable-server="true" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130628 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 13:30:23.136180 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130633 2559 flags.go:64] FLAG: --event-burst="100" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130636 2559 flags.go:64] FLAG: --event-qps="50" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130639 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130642 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130645 2559 flags.go:64] FLAG: --eviction-hard="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130648 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130651 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130654 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130658 2559 flags.go:64] FLAG: --eviction-soft="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130661 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130663 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130666 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130669 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130672 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130675 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130678 2559 flags.go:64] FLAG: --feature-gates="" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130681 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130686 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130689 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130692 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130695 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130698 2559 flags.go:64] FLAG: --help="false" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130701 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130704 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 13:30:23.136850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130707 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130709 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130713 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130716 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130719 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130722 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130724 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130727 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130731 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130734 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130737 2559 flags.go:64] FLAG: --kube-reserved="" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130740 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130743 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130745 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130748 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130751 2559 flags.go:64] FLAG: --lock-file="" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130754 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130757 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130760 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130766 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130768 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130771 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130774 2559 flags.go:64] FLAG: --logging-format="text" Sep 10 13:30:23.137504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130777 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130780 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130784 2559 flags.go:64] FLAG: --manifest-url="" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130786 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130790 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130794 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130797 2559 flags.go:64] FLAG: --max-pods="110" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130800 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130803 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130806 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130809 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130813 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130816 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130819 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130826 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130829 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130832 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130835 2559 flags.go:64] FLAG: --pod-cidr="" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130838 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130843 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130845 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130849 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130852 2559 flags.go:64] FLAG: --port="10250" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130855 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 13:30:23.138059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130858 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01a9d2c23eaa3306b" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130861 2559 flags.go:64] FLAG: --qos-reserved="" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130864 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130867 2559 flags.go:64] FLAG: --register-node="true" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130870 2559 flags.go:64] FLAG: --register-with-taints="" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130873 2559 flags.go:64] FLAG: --registry-burst="10" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130876 2559 flags.go:64] FLAG: --registry-qps="5" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130879 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130882 2559 flags.go:64] FLAG: --reserved-memory="" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130885 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130888 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130892 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130895 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130898 2559 flags.go:64] FLAG: --runonce="false" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130901 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130904 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130907 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130909 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130912 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130915 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130918 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130921 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130923 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130926 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130931 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130934 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 13:30:23.138659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130937 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130940 2559 flags.go:64] FLAG: --system-cgroups="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130943 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130948 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130951 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130953 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130958 2559 flags.go:64] FLAG: --tls-min-version="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130961 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130963 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130966 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130969 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130972 2559 flags.go:64] FLAG: --v="2" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130976 2559 flags.go:64] FLAG: --version="false" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130980 2559 flags.go:64] FLAG: --vmodule="" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130984 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.130987 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131105 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131109 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131112 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131115 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131118 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131122 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131124 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.139291 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131127 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131130 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131132 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131136 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131140 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131143 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131146 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131150 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131153 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131156 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131159 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131162 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131165 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131168 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131170 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131173 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131176 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131179 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131181 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131184 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.139846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131187 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131190 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131193 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131195 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131198 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131200 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131203 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131205 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131208 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131210 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131213 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131216 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131218 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131222 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131225 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131228 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131231 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131233 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131236 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131240 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.140340 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131243 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131246 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131248 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131251 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131253 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131256 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131259 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131262 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131265 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131268 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131271 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131273 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131276 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131279 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131281 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131284 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131288 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131292 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131295 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.140888 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131297 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131300 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131303 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131306 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131308 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131311 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131313 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131316 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131318 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131321 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131324 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131327 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131331 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131333 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131336 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131338 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131341 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131343 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131346 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131348 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.141357 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131351 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131353 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131356 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131358 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131361 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131363 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131366 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131369 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131371 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131374 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131378 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131380 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.131383 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.132096 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.139311 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 13:30:23.141837 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.139324 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.140538 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.140545 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140601 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140605 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140609 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140612 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140616 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140619 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140622 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140625 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140628 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140631 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140634 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140638 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140642 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140645 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140648 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140651 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.142231 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140654 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140657 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140660 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140662 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140665 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140668 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140670 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140673 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140676 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140679 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140689 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140692 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140695 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140699 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140702 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140705 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140707 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140710 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140713 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140715 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.142714 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140718 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140721 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140724 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140728 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140730 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140734 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140736 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140739 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140742 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140745 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140747 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140750 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140753 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140755 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140758 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140761 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140764 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140768 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140771 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.143202 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140773 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140776 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140780 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140782 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140785 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140787 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140790 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140795 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140798 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140800 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140803 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140806 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140808 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140811 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140813 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140816 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140818 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140821 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140824 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.143653 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140826 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140829 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140832 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140834 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140837 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140839 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140842 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140844 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140847 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140850 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140852 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140854 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140857 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140860 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140862 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140865 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140867 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140870 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140873 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.144183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140876 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140878 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140882 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140884 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140887 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.140890 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.140895 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141001 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141006 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141010 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141013 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141016 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141020 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141022 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 13:30:23.144632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141025 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141029 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141032 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141035 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141037 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141040 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141043 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141046 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141048 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141051 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141053 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141056 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141058 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141061 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141063 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141066 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141079 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141081 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141085 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 13:30:23.144985 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141087 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141090 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141093 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141096 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141099 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141101 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141104 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141106 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141109 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141111 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141114 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141118 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141121 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141124 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141127 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141129 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141132 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141134 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141137 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141139 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 13:30:23.145466 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141142 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141144 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141146 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141149 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141151 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141154 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141156 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141159 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141161 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141164 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141166 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141169 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141172 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141174 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141176 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141179 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141182 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141185 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141187 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141189 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 13:30:23.145957 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141192 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141194 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141197 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141199 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141202 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141204 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141207 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141209 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141212 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141215 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141217 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141220 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141222 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141225 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141227 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141230 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141232 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141234 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141237 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141239 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 13:30:23.146529 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141242 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141244 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141247 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141249 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141253 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141257 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141260 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141262 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141266 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141268 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141271 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141273 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:23.141276 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.141281 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 13:30:23.147014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.141372 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 13:30:23.147378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.144095 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 13:30:23.147378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.145028 2559 server.go:1019] "Starting client certificate rotation" Sep 10 13:30:23.147378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.145129 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:30:23.147378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.145883 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 13:30:23.147724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.147710 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:30:23.152512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.152490 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 13:30:23.161806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.161790 2559 log.go:25] "Validated CRI v1 runtime API" Sep 10 13:30:23.166553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.166538 2559 log.go:25] "Validated CRI v1 image API" Sep 10 13:30:23.167418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.167401 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 13:30:23.169986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.169968 2559 fs.go:135] Filesystem UUIDs: map[5473b21c-de9f-4763-8892-a9385f2f7d29:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dce7ec67-beae-4a4f-9f01-3a3e3ffcb773:/dev/nvme0n1p4] Sep 10 13:30:23.170046 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.169985 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 13:30:23.175661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.175551 2559 manager.go:216] Machine: {Timestamp:2026-09-10 13:30:23.174209816 +0000 UTC m=+0.369290433 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3069250 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cdb3afe04708740878ec510371274 SystemUUID:ec2cdb3a-fe04-7087-4087-8ec510371274 BootID:3d354907-666a-48b8-9934-242b36028b52 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9e:b0:1f:fb:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9e:b0:1f:fb:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:23:ac:be:ca:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 13:30:23.175661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.175660 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 13:30:23.175768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.175755 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 13:30:23.179407 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.179382 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 13:30:23.179539 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.179412 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-208.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 13:30:23.180217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.180203 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 13:30:23.180254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.180222 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 13:30:23.180254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.180232 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:30:23.180544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.180524 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:30:23.181108 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.181093 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 13:30:23.182262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.182252 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 13:30:23.187502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187488 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:30:23.187605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187596 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 13:30:23.187636 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187623 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 10 13:30:23.187665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187638 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 13:30:23.187665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187651 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 13:30:23.187665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187660 2559 kubelet.go:398] "Adding apiserver pod source" Sep 10 13:30:23.187779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187669 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 13:30:23.187976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187962 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:30:23.188019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.187990 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 13:30:23.190483 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.190464 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 13:30:23.194384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.194364 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 13:30:23.194464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.194407 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 13:30:23.196238 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196215 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196243 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196252 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196260 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196273 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196282 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196302 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 13:30:23.196326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196315 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 13:30:23.196531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196336 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 13:30:23.196531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196345 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 13:30:23.196531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196358 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 13:30:23.196531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.196381 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 13:30:23.197573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.197559 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 13:30:23.197620 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.197577 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 13:30:23.199396 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.199363 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 13:30:23.199396 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.199369 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 13:30:23.201249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.201236 2559 server.go:1268] "Started kubelet" Sep 10 13:30:23.201381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.201332 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 13:30:23.201431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.201345 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 13:30:23.201431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.201413 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 13:30:23.201891 ip-10-0-133-208 systemd[1]: Started Kubernetes Kubelet. Sep 10 13:30:23.202913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.202896 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 13:30:23.203472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.203447 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 10 13:30:23.207298 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.207278 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 13:30:23.209542 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.209525 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 13:30:23.209788 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.209773 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-208.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 13:30:23.210311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.210297 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 13:30:23.210395 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.210381 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.210984 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.210967 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 13:30:23.210984 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.210970 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.210994 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211042 2559 factory.go:55] Registering systemd factory Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211056 2559 factory.go:223] Registration of the systemd container factory successfully Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211107 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.209983 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f060c345 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.201207109 +0000 UTC m=+0.396287724,LastTimestamp:2026-09-10 13:30:23.201207109 +0000 UTC m=+0.396287724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.211136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211118 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 10 13:30:23.211420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211383 2559 factory.go:153] Registering CRI-O factory Sep 10 13:30:23.211420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211397 2559 factory.go:223] Registration of the crio container factory successfully Sep 10 13:30:23.211513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211441 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 13:30:23.211513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211471 2559 factory.go:103] Registering Raw factory Sep 10 13:30:23.211513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211487 2559 manager.go:1196] Started watching for new ooms in manager Sep 10 13:30:23.211986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.211972 2559 manager.go:349] Starting recovery of all containers Sep 10 13:30:23.214221 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.214172 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 13:30:23.214296 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.214258 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 13:30:23.224988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.224971 2559 manager.go:354] Recovery completed Sep 10 13:30:23.226318 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.226300 2559 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 10 13:30:23.229810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.229796 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.231566 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.231549 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.231644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.231578 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.231644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.231588 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.232163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232146 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 13:30:23.232163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232159 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.232294 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232161 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 13:30:23.232294 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232189 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 10 13:30:23.232933 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232918 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.233017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232947 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.233017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.232967 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.234245 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.234166 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f22fff38 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.231565624 +0000 UTC m=+0.426646241,LastTimestamp:2026-09-10 13:30:23.231565624 +0000 UTC m=+0.426646241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.234352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.234340 2559 policy_none.go:49] "None policy: Start" Sep 10 13:30:23.234400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.234359 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 13:30:23.234400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.234368 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 13:30:23.236361 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.236349 2559 policy_none.go:47] "Start" Sep 10 13:30:23.245182 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.245113 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f2304160 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.23158256 +0000 UTC m=+0.426663178,LastTimestamp:2026-09-10 13:30:23.23158256 +0000 UTC m=+0.426663178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.255188 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.255120 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f23065f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.231591921 +0000 UTC m=+0.426672535,LastTimestamp:2026-09-10 13:30:23.231591921 +0000 UTC m=+0.426672535,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.266848 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.266775 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-208.ec2.internal.18d3f8c9f22fff38\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f22fff38 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.231565624 +0000 UTC m=+0.426646241,LastTimestamp:2026-09-10 13:30:23.232932102 +0000 UTC m=+0.428012721,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.279741 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.279682 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-208.ec2.internal.18d3f8c9f2304160\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f2304160 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.23158256 +0000 UTC m=+0.426663178,LastTimestamp:2026-09-10 13:30:23.232959828 +0000 UTC m=+0.428040458,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.282615 2559 manager.go:337] "Starting Device Plugin manager" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.282644 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.282662 2559 server.go:87] "Starting device plugin registration server" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.282906 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.282919 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.283110 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.283182 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.283190 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.283580 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.283637 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.300479 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.293917 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-208.ec2.internal.18d3f8c9f23065f1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f23065f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-208.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.231591921 +0000 UTC m=+0.426672535,LastTimestamp:2026-09-10 13:30:23.232972897 +0000 UTC m=+0.428053517,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.306988 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.306926 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-208.ec2.internal.18d3f8c9f561252c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-208.ec2.internal,UID:ip-10-0-133-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-208.ec2.internal,},FirstTimestamp:2026-09-10 13:30:23.285118252 +0000 UTC m=+0.480198854,LastTimestamp:2026-09-10 13:30:23.285118252 +0000 UTC m=+0.480198854,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-208.ec2.internal,}" Sep 10 13:30:23.324780 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.324755 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 13:30:23.325908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.325886 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 13:30:23.325992 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.325943 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 13:30:23.325992 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.325972 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 13:30:23.326089 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.326014 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 13:30:23.334842 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.334824 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 10 13:30:23.350061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.350044 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqgg5" Sep 10 13:30:23.357942 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.357928 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqgg5" Sep 10 13:30:23.383335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.383278 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.384032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.384019 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.384118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.384046 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.384118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.384060 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.384118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.384102 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.401979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.401961 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.401979 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.401980 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-208.ec2.internal\": node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.425223 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.425204 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.426226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.426202 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal"] Sep 10 13:30:23.426308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.426259 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.427670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.427655 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.427751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.427687 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.427751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.427700 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.429931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.429919 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.430047 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.430032 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.430119 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.430065 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.430609 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.430593 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.430672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.430617 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.430672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.430633 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.431163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.431150 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.431217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.431173 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.431217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.431181 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.432847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.432833 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.432908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.432860 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:23.433513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.433499 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:23.433563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.433523 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:23.433563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.433533 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:23.466564 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.466545 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-208.ec2.internal\" not found" node="ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.470921 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.470908 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-208.ec2.internal\" not found" node="ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.512399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.512380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.512454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.512403 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a5b2711db4d2239ddb842b9e3de4842-config\") pod \"kube-apiserver-proxy-ip-10-0-133-208.ec2.internal\" (UID: \"4a5b2711db4d2239ddb842b9e3de4842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.512454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.512419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.525280 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.525256 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.613061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.613599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.613599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a5b2711db4d2239ddb842b9e3de4842-config\") pod \"kube-apiserver-proxy-ip-10-0-133-208.ec2.internal\" (UID: \"4a5b2711db4d2239ddb842b9e3de4842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.613599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4a5b2711db4d2239ddb842b9e3de4842-config\") pod \"kube-apiserver-proxy-ip-10-0-133-208.ec2.internal\" (UID: \"4a5b2711db4d2239ddb842b9e3de4842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.613599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.614099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.613442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e8410fd497a7c97f89188c03e217d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal\" (UID: \"5e8410fd497a7c97f89188c03e217d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.626163 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.626136 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.726959 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.726904 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.769081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.769059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.774947 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:23.774929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:23.827201 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.827178 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:23.927704 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:23.927667 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:25.613487 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:25.613465 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:25.713917 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:25.713894 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:25.730084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:25.730057 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 13:30:25.814659 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:25.814635 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:25.905683 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:25.905500 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:30:25.914837 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:25.914818 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:25.947977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:25.947950 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:25:24 +0000 UTC" deadline="2028-05-22 22:18:40.11811579 +0000 UTC" Sep 10 13:30:25.947977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:25.947976 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14888h48m14.170142649s" Sep 10 13:30:26.015461 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.015438 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:26.021597 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.021582 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 13:30:26.022567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.022551 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientMemory" Sep 10 13:30:26.022656 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.022584 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 13:30:26.022656 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.022601 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeHasSufficientPID" Sep 10 13:30:26.088889 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:26.088867 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5b2711db4d2239ddb842b9e3de4842.slice/crio-178e588982d05a2f7ca164ea0dc2950d59601eab7c5d1695c0b0ef8db3334114 WatchSource:0}: Error finding container 178e588982d05a2f7ca164ea0dc2950d59601eab7c5d1695c0b0ef8db3334114: Status 404 returned error can't find the container with id 178e588982d05a2f7ca164ea0dc2950d59601eab7c5d1695c0b0ef8db3334114 Sep 10 13:30:26.089300 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:26.089282 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8410fd497a7c97f89188c03e217d3d.slice/crio-22cf89d863f3db729248a28d5b89ccf5358a4cde9cdf0f0e5a2bd54f99635b74 WatchSource:0}: Error finding container 22cf89d863f3db729248a28d5b89ccf5358a4cde9cdf0f0e5a2bd54f99635b74: Status 404 returned error can't find the container with id 22cf89d863f3db729248a28d5b89ccf5358a4cde9cdf0f0e5a2bd54f99635b74 Sep 10 13:30:26.094101 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.094084 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:30:26.116332 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.116305 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-208.ec2.internal\" not found" Sep 10 13:30:26.133974 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.133953 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:30:26.176799 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.176755 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:30:26.195876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.195855 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" Sep 10 13:30:26.209241 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.209224 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:30:26.210075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.210061 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" Sep 10 13:30:26.219140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.219124 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 13:30:26.512871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.512809 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 13:30:26.774226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.774163 2559 apiserver.go:52] "Watching apiserver" Sep 10 13:30:26.780796 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.780780 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 13:30:26.781128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.781110 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-nkq6c","openshift-ovn-kubernetes/ovnkube-node-qrztv","kube-system/global-pull-secret-syncer-kh6q9","kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9","openshift-cluster-node-tuning-operator/tuned-l5dv6","openshift-multus/multus-additional-cni-plugins-k8x94","openshift-multus/network-metrics-daemon-c5z5r","kube-system/konnectivity-agent-mx2sk","openshift-dns/node-resolver-wzz2s","openshift-image-registry/node-ca-dxlmv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal","openshift-multus/multus-5lwqj","openshift-network-diagnostics/network-check-target-9htdb"] Sep 10 13:30:26.783616 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.783595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.786046 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.785808 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.786141 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.786095 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rm9b4\"" Sep 10 13:30:26.786204 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.786189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.786274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.786258 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 13:30:26.786322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.786311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.788941 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.788802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.788941 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.788814 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 13:30:26.789087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.788940 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.789593 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.789574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rzqq\"" Sep 10 13:30:26.791221 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.789991 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.791313 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.791251 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.791834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.791809 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.791923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.791878 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zr66c\"" Sep 10 13:30:26.792782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.792761 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.792891 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.792837 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:26.794874 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.794855 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.795832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.795805 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 13:30:26.797078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.797033 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.797258 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.797241 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.797642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.797283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.797642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.797410 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2vwk7\"" Sep 10 13:30:26.799290 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:26.799407 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.799337 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:26.799593 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799572 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 13:30:26.799685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799649 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 13:30:26.799773 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799757 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nv7wk\"" Sep 10 13:30:26.799830 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799785 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.799880 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.799846 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 13:30:26.800030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.800014 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.801727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.801707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:26.803791 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.803776 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 13:30:26.803927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.803779 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jjwnn\"" Sep 10 13:30:26.803927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.803899 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 13:30:26.804133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.804119 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:26.805259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jgq\" (UniqueName: \"kubernetes.io/projected/5828c67b-c93e-4af9-946e-2da852db81e3-kube-api-access-w2jgq\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.805353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cnibin\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.805353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-socket-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.805353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-sys-fs\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.805512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5828c67b-c93e-4af9-946e-2da852db81e3-hosts-file\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.805512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-modprobe-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-run\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-var-lib-kubelet\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805549 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdxv\" (UniqueName: \"kubernetes.io/projected/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kube-api-access-dmdxv\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-systemd\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-host\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.805706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805668 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-system-cni-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-registration-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-tuned\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805863 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-lib-modules\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-device-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r285h\" (UniqueName: \"kubernetes.io/projected/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-kube-api-access-r285h\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-kubelet-config\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.805983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysconfig\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-conf\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806042 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9k8\" (UniqueName: \"kubernetes.io/projected/fd7ad402-4d5f-4cd8-961d-61286e9a9678-kube-api-access-rh9k8\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-kubernetes\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-sys\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806197 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5828c67b-c93e-4af9-946e-2da852db81e3-tmp-dir\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806227 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-dbus\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-tmp\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxxc\" (UniqueName: \"kubernetes.io/projected/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-kube-api-access-lqxxc\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-host\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-serviceca\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806518 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.806627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806631 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mbdcr\"" Sep 10 13:30:26.807272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.806734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.808067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.808048 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 13:30:26.808798 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.808757 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkgg9\"" Sep 10 13:30:26.808989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.808952 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 13:30:26.809289 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 13:30:26.809392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809300 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.809770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809751 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 13:30:26.809853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809756 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 13:30:26.809906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 13:30:26.809952 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.809943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 13:30:26.811614 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.811596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:26.811703 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.811649 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:26.812131 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.812113 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 13:30:26.812131 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.812122 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dzw42\"" Sep 10 13:30:26.834002 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.833968 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c525v" Sep 10 13:30:26.843861 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.843842 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c525v" Sep 10 13:30:26.897575 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.897558 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 13:30:26.906600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906578 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-netd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.906691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-ovn\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.906691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-hostroot\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.906691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-etc-kubernetes\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.906691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/013592d0-e3a8-410f-9280-6b2b15e72391-host-slash\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5828c67b-c93e-4af9-946e-2da852db81e3-tmp-dir\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-dbus\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-tmp\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6qp\" (UniqueName: \"kubernetes.io/projected/22a2afc9-2500-44a2-9f17-b886c4ec7142-kube-api-access-5t6qp\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-host\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.906898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jgq\" (UniqueName: \"kubernetes.io/projected/5828c67b-c93e-4af9-946e-2da852db81e3-kube-api-access-w2jgq\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cnibin\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-netns\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906971 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-system-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.906994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cni-binary-copy\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-netns\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-socket-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-modprobe-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5828c67b-c93e-4af9-946e-2da852db81e3-tmp-dir\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-var-lib-kubelet\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-bin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-host\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdxv\" (UniqueName: \"kubernetes.io/projected/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kube-api-access-dmdxv\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-systemd\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.907245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907249 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/824f69e2-4b2e-44c9-a072-aa093b3f7b92-konnectivity-ca\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/013592d0-e3a8-410f-9280-6b2b15e72391-iptables-alerter-script\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-registration-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907388 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-dbus\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-k8s-cni-cncf-io\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-socket-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-modprobe-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-var-lib-kubelet\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907652 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-systemd\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cnibin\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-d\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-registration-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907705 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-conf-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qjs\" (UniqueName: \"kubernetes.io/projected/69b37c37-779c-4417-896d-a7b7c4f8d6ff-kube-api-access-98qjs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907801 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-lib-modules\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.908665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-device-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r285h\" (UniqueName: \"kubernetes.io/projected/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-kube-api-access-r285h\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysconfig\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907871 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-lib-modules\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-conf\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907932 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysconfig\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-sysctl-conf\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.907995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-device-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9k8\" (UniqueName: \"kubernetes.io/projected/fd7ad402-4d5f-4cd8-961d-61286e9a9678-kube-api-access-rh9k8\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-log-socket\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-kubernetes\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-sys\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-kubelet\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-slash\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.909472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-kubernetes\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-sys\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-multus\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-daemon-config\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxxc\" (UniqueName: \"kubernetes.io/projected/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-kube-api-access-lqxxc\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-var-lib-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.908471 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-env-overrides\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:26.908544 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:27.408522444 +0000 UTC m=+3.018307353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cnibin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-serviceca\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-kubelet\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-systemd-units\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-config\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5lq\" (UniqueName: \"kubernetes.io/projected/013592d0-e3a8-410f-9280-6b2b15e72391-kube-api-access-rz5lq\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-sys-fs\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5828c67b-c93e-4af9-946e-2da852db81e3-hosts-file\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-run\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-systemd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908843 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-sys-fs\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d229afbb-768f-4023-a60e-93a5c09d9404-ovn-node-metrics-cert\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-host\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-system-cni-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-run\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5828c67b-c93e-4af9-946e-2da852db81e3-hosts-file\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.908987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-etc-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.910948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-socket-dir-parent\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909048 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-multus-certs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-tuned\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-node-log\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-script-lib\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd7ad402-4d5f-4cd8-961d-61286e9a9678-system-cni-dir\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4cx\" (UniqueName: \"kubernetes.io/projected/d229afbb-768f-4023-a60e-93a5c09d9404-kube-api-access-lm4cx\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909181 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/824f69e2-4b2e-44c9-a072-aa093b3f7b92-agent-certs\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-host\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-kubelet-config\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-bin\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909026 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-serviceca\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.911813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909840 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:26.912610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd7ad402-4d5f-4cd8-961d-61286e9a9678-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.912610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.909921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7954047b-1d2d-4a68-8df2-db357abdf494-kubelet-config\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:26.912610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.912096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-etc-tuned\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.912871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.912848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-tmp\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.918557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.918539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r285h\" (UniqueName: \"kubernetes.io/projected/80cbebcd-aa00-4d3a-98ec-0e86bb4275e7-kube-api-access-r285h\") pod \"node-ca-dxlmv\" (UID: \"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7\") " pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:26.918670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.918602 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" event={"ID":"5e8410fd497a7c97f89188c03e217d3d","Type":"ContainerStarted","Data":"22cf89d863f3db729248a28d5b89ccf5358a4cde9cdf0f0e5a2bd54f99635b74"} Sep 10 13:30:26.919222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.919183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9k8\" (UniqueName: \"kubernetes.io/projected/fd7ad402-4d5f-4cd8-961d-61286e9a9678-kube-api-access-rh9k8\") pod \"multus-additional-cni-plugins-k8x94\" (UID: \"fd7ad402-4d5f-4cd8-961d-61286e9a9678\") " pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:26.919429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.919405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jgq\" (UniqueName: \"kubernetes.io/projected/5828c67b-c93e-4af9-946e-2da852db81e3-kube-api-access-w2jgq\") pod \"node-resolver-wzz2s\" (UID: \"5828c67b-c93e-4af9-946e-2da852db81e3\") " pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:26.920479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.920453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxxc\" (UniqueName: \"kubernetes.io/projected/4cd37b4a-1eb2-4dcf-917e-76f4efd4f561-kube-api-access-lqxxc\") pod \"tuned-l5dv6\" (UID: \"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561\") " pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:26.921119 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.920914 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" event={"ID":"4a5b2711db4d2239ddb842b9e3de4842","Type":"ContainerStarted","Data":"178e588982d05a2f7ca164ea0dc2950d59601eab7c5d1695c0b0ef8db3334114"} Sep 10 13:30:26.921598 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:26.921536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdxv\" (UniqueName: \"kubernetes.io/projected/8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0-kube-api-access-dmdxv\") pod \"aws-ebs-csi-driver-node-zmtk9\" (UID: \"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:27.009530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-log-socket\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-kubelet\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-slash\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-multus\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-daemon-config\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009652 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.009677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-var-lib-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-log-socket\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-env-overrides\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009742 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cnibin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-kubelet\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-kubelet\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-systemd-units\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.009854 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-systemd-units\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-multus\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.009897 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:27.509885418 +0000 UTC m=+3.119670309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.009921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-slash\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cnibin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-var-lib-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-kubelet\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-config\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5lq\" (UniqueName: \"kubernetes.io/projected/013592d0-e3a8-410f-9280-6b2b15e72391-kube-api-access-rz5lq\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-systemd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d229afbb-768f-4023-a60e-93a5c09d9404-ovn-node-metrics-cert\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-etc-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-socket-dir-parent\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-systemd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-multus-certs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-node-log\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-script-lib\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4cx\" (UniqueName: \"kubernetes.io/projected/d229afbb-768f-4023-a60e-93a5c09d9404-kube-api-access-lm4cx\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/824f69e2-4b2e-44c9-a072-aa093b3f7b92-agent-certs\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-bin\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-etc-openvswitch\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-node-log\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-daemon-config\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-multus-certs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-netd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.010888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-netd\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-cni-bin\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-ovn\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-env-overrides\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-hostroot\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-run-ovn\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-etc-kubernetes\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-hostroot\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/013592d0-e3a8-410f-9280-6b2b15e72391-host-slash\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-config\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-socket-dir-parent\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010672 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-etc-kubernetes\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6qp\" (UniqueName: \"kubernetes.io/projected/22a2afc9-2500-44a2-9f17-b886c4ec7142-kube-api-access-5t6qp\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010713 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/013592d0-e3a8-410f-9280-6b2b15e72391-host-slash\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010715 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-netns\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-run-netns\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-system-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.011767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cni-binary-copy\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-netns\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-system-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-bin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/824f69e2-4b2e-44c9-a072-aa093b3f7b92-konnectivity-ca\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-cni-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/013592d0-e3a8-410f-9280-6b2b15e72391-iptables-alerter-script\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-var-lib-cni-bin\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-k8s-cni-cncf-io\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-netns\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.010982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-conf-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011010 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d229afbb-768f-4023-a60e-93a5c09d9404-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011011 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qjs\" (UniqueName: \"kubernetes.io/projected/69b37c37-779c-4417-896d-a7b7c4f8d6ff-kube-api-access-98qjs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d229afbb-768f-4023-a60e-93a5c09d9404-ovnkube-script-lib\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-multus-conf-dir\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69b37c37-779c-4417-896d-a7b7c4f8d6ff-host-run-k8s-cni-cncf-io\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b37c37-779c-4417-896d-a7b7c4f8d6ff-cni-binary-copy\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.012726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.011477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/824f69e2-4b2e-44c9-a072-aa093b3f7b92-konnectivity-ca\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:27.013621 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.012198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/013592d0-e3a8-410f-9280-6b2b15e72391-iptables-alerter-script\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.013621 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.013503 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/824f69e2-4b2e-44c9-a072-aa093b3f7b92-agent-certs\") pod \"konnectivity-agent-mx2sk\" (UID: \"824f69e2-4b2e-44c9-a072-aa093b3f7b92\") " pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:27.013758 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.013737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d229afbb-768f-4023-a60e-93a5c09d9404-ovn-node-metrics-cert\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.017928 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.017907 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:27.018018 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.017933 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:27.018018 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.017945 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:27.018018 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.018008 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:27.517993547 +0000 UTC m=+3.127778450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:27.020584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.020563 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4cx\" (UniqueName: \"kubernetes.io/projected/d229afbb-768f-4023-a60e-93a5c09d9404-kube-api-access-lm4cx\") pod \"ovnkube-node-qrztv\" (UID: \"d229afbb-768f-4023-a60e-93a5c09d9404\") " pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.021022 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.021001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5lq\" (UniqueName: \"kubernetes.io/projected/013592d0-e3a8-410f-9280-6b2b15e72391-kube-api-access-rz5lq\") pod \"iptables-alerter-nkq6c\" (UID: \"013592d0-e3a8-410f-9280-6b2b15e72391\") " pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.022476 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.022434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qjs\" (UniqueName: \"kubernetes.io/projected/69b37c37-779c-4417-896d-a7b7c4f8d6ff-kube-api-access-98qjs\") pod \"multus-5lwqj\" (UID: \"69b37c37-779c-4417-896d-a7b7c4f8d6ff\") " pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.022628 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.022604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6qp\" (UniqueName: \"kubernetes.io/projected/22a2afc9-2500-44a2-9f17-b886c4ec7142-kube-api-access-5t6qp\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:27.096779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.096708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" Sep 10 13:30:27.102451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.102431 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxlmv" Sep 10 13:30:27.112132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.111938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wzz2s" Sep 10 13:30:27.116635 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.116615 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" Sep 10 13:30:27.123640 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.123608 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5828c67b_c93e_4af9_946e_2da852db81e3.slice/crio-90514ff5ff3de377d27aa77e48783b57557fa02a3c6fae93b606a3fabf969ff3 WatchSource:0}: Error finding container 90514ff5ff3de377d27aa77e48783b57557fa02a3c6fae93b606a3fabf969ff3: Status 404 returned error can't find the container with id 90514ff5ff3de377d27aa77e48783b57557fa02a3c6fae93b606a3fabf969ff3 Sep 10 13:30:27.124345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.124323 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8x94" Sep 10 13:30:27.126807 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.126734 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd37b4a_1eb2_4dcf_917e_76f4efd4f561.slice/crio-f796612cebecab310c411b4918582b35eb8612d25e345cecbcdd5a5ca1284f40 WatchSource:0}: Error finding container f796612cebecab310c411b4918582b35eb8612d25e345cecbcdd5a5ca1284f40: Status 404 returned error can't find the container with id f796612cebecab310c411b4918582b35eb8612d25e345cecbcdd5a5ca1284f40 Sep 10 13:30:27.132399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.132360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:27.135873 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.135851 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7ad402_4d5f_4cd8_961d_61286e9a9678.slice/crio-cde01a504985b13b0321632ec4991b02a8cd542bf1b595ba937046734f581496 WatchSource:0}: Error finding container cde01a504985b13b0321632ec4991b02a8cd542bf1b595ba937046734f581496: Status 404 returned error can't find the container with id cde01a504985b13b0321632ec4991b02a8cd542bf1b595ba937046734f581496 Sep 10 13:30:27.140918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.140897 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nkq6c" Sep 10 13:30:27.142780 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.142760 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824f69e2_4b2e_44c9_a072_aa093b3f7b92.slice/crio-3c91444b9b818913779dcae3d331babe41ccc12ad5a88fa1ef86db468f5c1603 WatchSource:0}: Error finding container 3c91444b9b818913779dcae3d331babe41ccc12ad5a88fa1ef86db468f5c1603: Status 404 returned error can't find the container with id 3c91444b9b818913779dcae3d331babe41ccc12ad5a88fa1ef86db468f5c1603 Sep 10 13:30:27.147894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.147873 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:27.150913 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.150869 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013592d0_e3a8_410f_9280_6b2b15e72391.slice/crio-7653d27fa8f02fb4f8d5f5f5474fcc69505094137411ee8efbd190ad861d3d72 WatchSource:0}: Error finding container 7653d27fa8f02fb4f8d5f5f5474fcc69505094137411ee8efbd190ad861d3d72: Status 404 returned error can't find the container with id 7653d27fa8f02fb4f8d5f5f5474fcc69505094137411ee8efbd190ad861d3d72 Sep 10 13:30:27.154602 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.154580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lwqj" Sep 10 13:30:27.159783 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.159762 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd229afbb_768f_4023_a60e_93a5c09d9404.slice/crio-16afe9a1b0acd5cf34e7a4edb88e76fa90bf7c4c68821f216ba06659e046b272 WatchSource:0}: Error finding container 16afe9a1b0acd5cf34e7a4edb88e76fa90bf7c4c68821f216ba06659e046b272: Status 404 returned error can't find the container with id 16afe9a1b0acd5cf34e7a4edb88e76fa90bf7c4c68821f216ba06659e046b272 Sep 10 13:30:27.167495 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:27.167469 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b37c37_779c_4417_896d_a7b7c4f8d6ff.slice/crio-1fd8df060fd51a26a261a63d227bb50bd8181773757fae54454539b702207851 WatchSource:0}: Error finding container 1fd8df060fd51a26a261a63d227bb50bd8181773757fae54454539b702207851: Status 404 returned error can't find the container with id 1fd8df060fd51a26a261a63d227bb50bd8181773757fae54454539b702207851 Sep 10 13:30:27.414282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.414209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:27.414447 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.414350 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:27.414447 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.414438 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:28.414417886 +0000 UTC m=+4.024202778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:27.514641 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.514605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:27.514817 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.514726 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:27.514817 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.514800 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:28.514781862 +0000 UTC m=+4.124566754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:27.615755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.615718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:27.615936 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.615913 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:27.615989 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.615944 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:27.615989 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.615959 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:27.616053 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.616023 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:28.616003229 +0000 UTC m=+4.225788124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:27.847506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.847433 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:25:26 +0000 UTC" deadline="2028-04-14 16:57:33.185678093 +0000 UTC" Sep 10 13:30:27.847506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.847460 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13971h27m5.338221049s" Sep 10 13:30:27.911988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.911961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:27.912131 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:27.912088 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:27.924726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.924644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nkq6c" event={"ID":"013592d0-e3a8-410f-9280-6b2b15e72391","Type":"ContainerStarted","Data":"7653d27fa8f02fb4f8d5f5f5474fcc69505094137411ee8efbd190ad861d3d72"} Sep 10 13:30:27.926126 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.926101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mx2sk" event={"ID":"824f69e2-4b2e-44c9-a072-aa093b3f7b92","Type":"ContainerStarted","Data":"3c91444b9b818913779dcae3d331babe41ccc12ad5a88fa1ef86db468f5c1603"} Sep 10 13:30:27.927955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.927928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" event={"ID":"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561","Type":"ContainerStarted","Data":"f796612cebecab310c411b4918582b35eb8612d25e345cecbcdd5a5ca1284f40"} Sep 10 13:30:27.929357 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.929312 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wzz2s" event={"ID":"5828c67b-c93e-4af9-946e-2da852db81e3","Type":"ContainerStarted","Data":"90514ff5ff3de377d27aa77e48783b57557fa02a3c6fae93b606a3fabf969ff3"} Sep 10 13:30:27.930557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.930521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxlmv" event={"ID":"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7","Type":"ContainerStarted","Data":"9dac4e7728688ffaf2948ca732d905ca2f89fa81c98d9ea5e6075b5f4f558008"} Sep 10 13:30:27.931836 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.931808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" event={"ID":"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0","Type":"ContainerStarted","Data":"6ace806337ca184eb38d61efc8db4595b427989908df607cb67c860b27a31ae1"} Sep 10 13:30:27.933299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.933275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lwqj" event={"ID":"69b37c37-779c-4417-896d-a7b7c4f8d6ff","Type":"ContainerStarted","Data":"1fd8df060fd51a26a261a63d227bb50bd8181773757fae54454539b702207851"} Sep 10 13:30:27.934905 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.934869 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerStarted","Data":"cde01a504985b13b0321632ec4991b02a8cd542bf1b595ba937046734f581496"} Sep 10 13:30:27.936329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:27.936300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"16afe9a1b0acd5cf34e7a4edb88e76fa90bf7c4c68821f216ba06659e046b272"} Sep 10 13:30:28.422208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.422179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:28.422392 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.422344 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:28.422451 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.422433 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:30.422413814 +0000 UTC m=+6.032198710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:28.523670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.523594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:28.523895 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.523774 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:28.523895 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.523839 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:30.523821317 +0000 UTC m=+6.133606215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:28.624030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.623997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:28.624198 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.624179 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:28.624291 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.624205 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:28.624346 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.624220 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:28.624417 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.624393 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:30.624359266 +0000 UTC m=+6.234144160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:28.848577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.848421 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:25:26 +0000 UTC" deadline="2028-02-05 08:16:49.474458093 +0000 UTC" Sep 10 13:30:28.848577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.848458 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12306h46m20.626004362s" Sep 10 13:30:28.914848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.914207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:28.914848 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.914320 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:28.914848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.914703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:28.914848 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:28.914806 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:28.949144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.949095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" event={"ID":"4a5b2711db4d2239ddb842b9e3de4842","Type":"ContainerStarted","Data":"bcfaeafbc0b150c918f0d1549f6a920bf05aacb8f1aa6478cbb761e47bc4087c"} Sep 10 13:30:28.967883 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:28.967631 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-208.ec2.internal" podStartSLOduration=2.967611135 podStartE2EDuration="2.967611135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:28.966498795 +0000 UTC m=+4.576283712" watchObservedRunningTime="2026-09-10 13:30:28.967611135 +0000 UTC m=+4.577396050" Sep 10 13:30:29.912138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:29.911790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:29.912627 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:29.912128 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:29.960959 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:29.960924 2559 generic.go:356] "Generic (PLEG): container finished" podID="5e8410fd497a7c97f89188c03e217d3d" containerID="4477f88ef99e022aa27ca564161f67cb221302e3056a459114ad9a52a5f178d8" exitCode=0 Sep 10 13:30:29.961722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:29.961444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" event={"ID":"5e8410fd497a7c97f89188c03e217d3d","Type":"ContainerDied","Data":"4477f88ef99e022aa27ca564161f67cb221302e3056a459114ad9a52a5f178d8"} Sep 10 13:30:30.441260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.441076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:30.441260 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.441238 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:30.441492 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.441308 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:34.441288672 +0000 UTC m=+10.051073577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:30.542647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.542065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:30.542647 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.542231 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:30.542647 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.542289 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:34.542271392 +0000 UTC m=+10.152056286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:30.643442 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.643327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:30.643591 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.643489 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:30.643591 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.643511 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:30.643591 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.643522 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:30.643591 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.643575 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:34.643557747 +0000 UTC m=+10.253342652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:30.913553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.913479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:30.913993 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.913619 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:30.914193 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.914174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:30.914296 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:30.914278 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:30.972106 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.972071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" event={"ID":"5e8410fd497a7c97f89188c03e217d3d","Type":"ContainerStarted","Data":"023e271dc02adefb9f02002f2bf82078d1894883c9ca9fc2fc5015ead712f8bc"} Sep 10 13:30:30.995446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:30.994358 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-208.ec2.internal" podStartSLOduration=4.994339727 podStartE2EDuration="4.994339727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:30.994232721 +0000 UTC m=+6.604017635" watchObservedRunningTime="2026-09-10 13:30:30.994339727 +0000 UTC m=+6.604124635" Sep 10 13:30:31.911967 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:31.911934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:31.912138 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:31.912063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:32.912401 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:32.912357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:32.912839 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:32.912359 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:32.912839 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:32.912495 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:32.912839 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:32.912605 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:33.912222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:33.912195 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:33.912427 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:33.912318 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:34.472050 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:34.472014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:34.472206 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.472157 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:34.472257 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.472222 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:42.472202355 +0000 UTC m=+18.081987260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:34.572632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:34.572601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:34.572810 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.572755 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:34.572865 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.572817 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:42.572802499 +0000 UTC m=+18.182587394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:34.673163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:34.673128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:34.673325 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.673298 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:34.673325 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.673317 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:34.673461 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.673330 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:34.673461 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.673401 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:42.673383372 +0000 UTC m=+18.283168277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:34.913337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:34.913271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:34.913735 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.913411 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:34.913735 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:34.913452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:34.913735 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:34.913599 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:35.912334 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:35.912299 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:35.912532 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:35.912438 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:36.911873 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:36.911843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:36.912314 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:36.911984 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:36.912420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:36.912326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:36.912478 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:36.912460 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:37.911654 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:37.911621 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:37.911814 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:37.911755 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:38.912042 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:38.912010 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:38.912519 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:38.912133 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:38.912519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:38.912188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:38.912519 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:38.912274 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:39.912169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:39.912141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:39.912652 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:39.912261 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:40.911617 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:40.911581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:40.911795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:40.911594 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:40.911795 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:40.911727 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:40.911882 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:40.911795 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:41.911954 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:41.911925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:41.912491 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:41.912034 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:42.529613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:42.529581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:42.529786 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.529722 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:42.529786 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.529784 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:58.52976455 +0000 UTC m=+34.139549455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:42.630727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:42.630697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:42.630874 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.630854 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:42.630934 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.630914 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:58.630896714 +0000 UTC m=+34.240681830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:42.731860 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:42.731822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:42.732001 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.731979 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 13:30:42.732047 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.732001 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 13:30:42.732047 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.732014 2559 projected.go:196] Error preparing data for projected volume kube-api-access-sg2hk for pod openshift-network-diagnostics/network-check-target-9htdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:42.732129 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.732073 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk podName:02dc4a55-e539-4eea-960e-46fb50b74ad5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:58.732056498 +0000 UTC m=+34.341841403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sg2hk" (UniqueName: "kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk") pod "network-check-target-9htdb" (UID: "02dc4a55-e539-4eea-960e-46fb50b74ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 13:30:42.912223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:42.912135 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:42.912675 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.912276 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:42.912675 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:42.912323 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:42.912675 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:42.912452 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:43.911777 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:43.911746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:43.911950 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:43.911860 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:44.914768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:44.914737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:44.915177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:44.914778 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:44.915177 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:44.914862 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:44.915177 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:44.914997 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:45.911520 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:45.911496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:45.911629 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:45.911610 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:46.000459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.000432 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="75a62a0ad416ac6d750424160bf0601a0a245d8985411222b57ebd8b3d47e6a0" exitCode=0 Sep 10 13:30:46.001133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.000508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"75a62a0ad416ac6d750424160bf0601a0a245d8985411222b57ebd8b3d47e6a0"} Sep 10 13:30:46.003705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.003682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"6d824008eaf8803be5fcc428a0a23065d4c824f775a50f9a8d74b38eb6d11cf8"} Sep 10 13:30:46.003809 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.003755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"952ac468c5c8d8d6a1195ad1f4ecacc44259443e4de887d3b0fe4c5f4fb745f0"} Sep 10 13:30:46.003809 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.003796 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"61353b21ec6859776a0e770f389c7785bd50f8e9fb665b88c1ef19b22cc89e15"} Sep 10 13:30:46.003898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.003828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"d916e86f899ac2c80ab0a6d5b19945118c2ea25af1d81e0e9e5302fe380de28a"} Sep 10 13:30:46.003898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.003859 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"172b31f7b6f43bfdce14519cae8344cee5cbfe22caee744049c3353e3b055376"} Sep 10 13:30:46.005525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.005492 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mx2sk" event={"ID":"824f69e2-4b2e-44c9-a072-aa093b3f7b92","Type":"ContainerStarted","Data":"2d6f64aebbb56fb077c03427199c6b819f0be5fe1d220ec62677c25389898986"} Sep 10 13:30:46.009118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.009084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" event={"ID":"4cd37b4a-1eb2-4dcf-917e-76f4efd4f561","Type":"ContainerStarted","Data":"23981c404451797e241e0ee4165194a7591e9511f9437b6e56497e37a15768b9"} Sep 10 13:30:46.012523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.012501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wzz2s" event={"ID":"5828c67b-c93e-4af9-946e-2da852db81e3","Type":"ContainerStarted","Data":"2e6ac4d6c527f23f9dd3010d8f6519b4f434c8bc614085866838cc344a06d80c"} Sep 10 13:30:46.014142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.014094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxlmv" event={"ID":"80cbebcd-aa00-4d3a-98ec-0e86bb4275e7","Type":"ContainerStarted","Data":"72830c486bfe883cbeacf3ec1fdbabfb72562707f721d7ab9c6cc4edf95a4a44"} Sep 10 13:30:46.015637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.015616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" event={"ID":"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0","Type":"ContainerStarted","Data":"68f668c78f0673e960ddf1edf0b8a90fc6d80a2865a185894c752caf023cb64e"} Sep 10 13:30:46.017056 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.017037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lwqj" event={"ID":"69b37c37-779c-4417-896d-a7b7c4f8d6ff","Type":"ContainerStarted","Data":"f52e96dd39b19465f697dbf869d70a037b6b9b6c8fd1d1a0a00ac5fd981639c8"} Sep 10 13:30:46.049886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.049850 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5lwqj" podStartSLOduration=2.937752407 podStartE2EDuration="21.049838821s" totalImagesPullingTime="18.112086414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:46.049689003 +0000 UTC m=+21.659473915" watchObservedRunningTime="2026-09-10 13:30:46.049838821 +0000 UTC m=+21.659623733" Sep 10 13:30:46.076017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.075867 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dxlmv" podStartSLOduration=3.94645598 podStartE2EDuration="22.075855757s" totalImagesPullingTime="18.129399777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:46.075069115 +0000 UTC m=+21.684854029" watchObservedRunningTime="2026-09-10 13:30:46.075855757 +0000 UTC m=+21.685640666" Sep 10 13:30:46.096613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.096580 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wzz2s" podStartSLOduration=2.978223528 podStartE2EDuration="21.096571009s" totalImagesPullingTime="18.118347481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:46.096491433 +0000 UTC m=+21.706276349" watchObservedRunningTime="2026-09-10 13:30:46.096571009 +0000 UTC m=+21.706355921" Sep 10 13:30:46.144680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.144640 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l5dv6" podStartSLOduration=3.025321788 podStartE2EDuration="21.144628222s" totalImagesPullingTime="18.119306434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:46.144413493 +0000 UTC m=+21.754198408" watchObservedRunningTime="2026-09-10 13:30:46.144628222 +0000 UTC m=+21.754413134" Sep 10 13:30:46.144868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.144843 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mx2sk" podStartSLOduration=3.045532518 podStartE2EDuration="21.144837145s" totalImagesPullingTime="18.099304627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:46.117802677 +0000 UTC m=+21.727587589" watchObservedRunningTime="2026-09-10 13:30:46.144837145 +0000 UTC m=+21.754622058" Sep 10 13:30:46.544320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.544248 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:46.545124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.545104 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:46.698475 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.698456 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 13:30:46.879537 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.879413 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:30:46.698471839Z","UUID":"9d0bc43f-3176-4d49-b9d3-56be7b461476","Handler":null,"Name":"","Endpoint":""} Sep 10 13:30:46.880677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.880660 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 13:30:46.880779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.880685 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 13:30:46.912329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.912303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:46.912473 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:46.912450 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:46.912540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:46.912483 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:46.912607 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:46.912587 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:47.022987 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.022952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"0d2ae84e6cbefada22e99e0743c3a9cbf08f6b555f6fd8c0c27168942e0cee2d"} Sep 10 13:30:47.024670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.024603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nkq6c" event={"ID":"013592d0-e3a8-410f-9280-6b2b15e72391","Type":"ContainerStarted","Data":"ed86d54f593e0b99ae508457e90ffbf030a1728f2b3373b552e27d362dba6141"} Sep 10 13:30:47.026496 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.026471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" event={"ID":"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0","Type":"ContainerStarted","Data":"a45dc3694dc7f824b7cc7ca22dc098982bd1d94debd03d35fe274067aa7cd150"} Sep 10 13:30:47.026829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.026745 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:47.030186 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.028541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mx2sk" Sep 10 13:30:47.076543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.076499 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nkq6c" podStartSLOduration=3.985488469 podStartE2EDuration="22.076486544s" totalImagesPullingTime="18.090998075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:47.043734779 +0000 UTC m=+22.653519693" watchObservedRunningTime="2026-09-10 13:30:47.076486544 +0000 UTC m=+22.686271456" Sep 10 13:30:47.912384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:47.912341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:47.912572 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:47.912496 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:48.912382 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:48.912350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:48.912784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:48.912350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:48.912784 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:48.912474 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:48.912784 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:48.912580 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:49.033890 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:49.033860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"a2d854ddce4bbe82a6d17b4a5124e94fdac2e23dde9a441eaf018c1c064eaf4a"} Sep 10 13:30:49.035850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:49.035824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" event={"ID":"8a13cc92-7c0d-44ac-a950-1e5b5d2e51c0","Type":"ContainerStarted","Data":"ee33343e43a1f82a0fcba8b29dc599318cd886e5cdfc23c36d41adc6f328a6fe"} Sep 10 13:30:49.058442 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:49.058397 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zmtk9" podStartSLOduration=4.251058513 podStartE2EDuration="25.058378839s" totalImagesPullingTime="20.807320326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:49.057884311 +0000 UTC m=+24.667669225" watchObservedRunningTime="2026-09-10 13:30:49.058378839 +0000 UTC m=+24.668163745" Sep 10 13:30:49.912319 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:49.912283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:49.912506 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:49.912414 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:50.911866 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:50.911627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:50.912046 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:50.911698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:50.912046 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:50.911989 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:50.912153 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:50.912084 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:51.911634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:51.911584 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:51.912279 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:51.911675 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:52.045214 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.045188 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="4023fb0ad294d2f5e24f1be176fc68fcd4b429caff268b9d79fd6f7f9274a82d" exitCode=0 Sep 10 13:30:52.045341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.045274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"4023fb0ad294d2f5e24f1be176fc68fcd4b429caff268b9d79fd6f7f9274a82d"} Sep 10 13:30:52.049183 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.049149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" event={"ID":"d229afbb-768f-4023-a60e-93a5c09d9404","Type":"ContainerStarted","Data":"b0d9c5febcf44c74afc7cb1049f6f13e53d959f36f2b69df56bf5a01ad8d846c"} Sep 10 13:30:52.049438 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.049418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:52.049534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.049501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:52.049667 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.049553 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:52.064002 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.063981 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:52.064422 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.064406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:30:52.152924 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.152881 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" podStartSLOduration=8.730695695 podStartE2EDuration="27.152869399s" totalImagesPullingTime="18.422173704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:52.152633838 +0000 UTC m=+27.762418752" watchObservedRunningTime="2026-09-10 13:30:52.152869399 +0000 UTC m=+27.762654340" Sep 10 13:30:52.911770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.911539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:52.912968 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:52.911549 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:52.912968 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:52.912556 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:52.913437 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:52.913360 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:53.208975 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.208872 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kh6q9"] Sep 10 13:30:53.209100 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.208993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:53.209100 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:53.209084 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:53.213614 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.213590 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5z5r"] Sep 10 13:30:53.213719 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.213682 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:53.213792 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:53.213774 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:53.216810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.216776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9htdb"] Sep 10 13:30:53.216909 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:53.216850 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:53.216951 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:53.216913 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:54.055863 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:54.055831 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="362b47a19deeb17bfefeda87d573a7cc040c0e2f1a66a5cfc167e571dc0394fa" exitCode=0 Sep 10 13:30:54.056319 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:54.055913 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"362b47a19deeb17bfefeda87d573a7cc040c0e2f1a66a5cfc167e571dc0394fa"} Sep 10 13:30:54.912400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:54.912350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:54.912553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:54.912412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:54.912553 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:54.912491 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:54.912553 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:54.912517 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:54.912553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:54.912549 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:54.912767 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:54.912608 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:56.061864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:56.061801 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="d8250201df19cd5345f63db5c277e6d4fd585d852c2258a79808411deb542a81" exitCode=0 Sep 10 13:30:56.061864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:56.061833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"d8250201df19cd5345f63db5c277e6d4fd585d852c2258a79808411deb542a81"} Sep 10 13:30:56.912177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:56.912106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:56.912177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:56.912105 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:56.912384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:56.912105 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:56.912384 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:56.912211 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9htdb" podUID="02dc4a55-e539-4eea-960e-46fb50b74ad5" Sep 10 13:30:56.912384 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:56.912298 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c5z5r" podUID="22a2afc9-2500-44a2-9f17-b886c4ec7142" Sep 10 13:30:56.912384 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:56.912350 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kh6q9" podUID="7954047b-1d2d-4a68-8df2-db357abdf494" Sep 10 13:30:58.186864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.186612 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-208.ec2.internal" event="NodeReady" Sep 10 13:30:58.187399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.186934 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 13:30:58.246855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.246831 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:30:58.269797 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.269775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.272271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.272246 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 13:30:58.272432 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.272282 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 13:30:58.272432 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.272304 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 13:30:58.272543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.272438 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tn4wt\"" Sep 10 13:30:58.278038 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.278014 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 13:30:58.280117 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.280095 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ccxnk"] Sep 10 13:30:58.287881 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.287862 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:30:58.287981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.287948 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.290222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.289725 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv"] Sep 10 13:30:58.290326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.290287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 13:30:58.290535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.290518 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 13:30:58.290732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.290712 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9764\"" Sep 10 13:30:58.303532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.303514 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4"] Sep 10 13:30:58.303662 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.303645 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.307564 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.307510 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.307564 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.307556 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 13:30:58.307723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.307600 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.307788 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.307771 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8kf4z\"" Sep 10 13:30:58.307841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.307789 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 13:30:58.320908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.320890 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-49wvj"] Sep 10 13:30:58.321010 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.320998 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.324222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.324173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.324222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.324184 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.324222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.324180 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dwcmz\"" Sep 10 13:30:58.324570 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.324551 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 13:30:58.324654 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.324619 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 13:30:58.345292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.345271 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s"] Sep 10 13:30:58.345417 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.345401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.347528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.347507 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 13:30:58.347669 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.347517 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jkzd4\"" Sep 10 13:30:58.347824 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.347807 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 13:30:58.350632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swtn\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.350895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.351052 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.350954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.355160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.355134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:30:58.367901 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.366574 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf"] Sep 10 13:30:58.368062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.368037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" Sep 10 13:30:58.370602 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.370579 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j8vj9\"" Sep 10 13:30:58.372560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.372539 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.372698 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.372576 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.382642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.382621 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgjcm"] Sep 10 13:30:58.382747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.382726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.385208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.385191 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.385342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.385325 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 13:30:58.385623 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.385605 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8g9kf\"" Sep 10 13:30:58.385712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.385664 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 13:30:58.385712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.385605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.400778 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.400760 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-676fd4f68-q9ztc"] Sep 10 13:30:58.400898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.400883 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.410223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.409945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 13:30:58.410223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.410045 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.410223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.410112 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 13:30:58.410223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.410191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gtwqn\"" Sep 10 13:30:58.415794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.415778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.420254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.420234 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 13:30:58.422821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.422802 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn"] Sep 10 13:30:58.422949 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.422933 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.433740 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.433683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 13:30:58.433941 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.433922 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 13:30:58.434024 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.433922 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.434316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.434302 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.434549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.434534 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 13:30:58.434644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.434581 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xzcxz\"" Sep 10 13:30:58.434644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.434599 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 13:30:58.434939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.434908 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n9twt"] Sep 10 13:30:58.435041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.435022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.444306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.444161 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.447794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.446957 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5dzcw\"" Sep 10 13:30:58.450735 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.450694 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.451089 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dkbfp"] Sep 10 13:30:58.451487 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.451571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swtn\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.451571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.451571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fr7\" (UniqueName: \"kubernetes.io/projected/9936248a-cfed-420b-914f-b5ebc63997ea-kube-api-access-q9fr7\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea039094-07d1-4d6b-98b1-04f6134ed357-config\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-nginx-conf\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea039094-07d1-4d6b-98b1-04f6134ed357-serving-cert\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.451729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kzp\" (UniqueName: \"kubernetes.io/projected/97c642c5-28bd-4ea9-97dd-311e2d943a9a-kube-api-access-v5kzp\") pod \"volume-data-source-validator-695c895bd6-l5l4s\" (UID: \"97c642c5-28bd-4ea9-97dd-311e2d943a9a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwcs\" (UniqueName: \"kubernetes.io/projected/ea039094-07d1-4d6b-98b1-04f6134ed357-kube-api-access-dzwcs\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9936248a-cfed-420b-914f-b5ebc63997ea-config-volume\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.451987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxzq\" (UniqueName: \"kubernetes.io/projected/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-kube-api-access-7wxzq\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.452191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.452029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9936248a-cfed-420b-914f-b5ebc63997ea-tmp-dir\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.452763 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.452417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.453050 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.453032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.453270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.453248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.454118 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.454099 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:30:58.454118 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.454121 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:30:58.454274 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.454169 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:58.954151249 +0000 UTC m=+34.563936147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:30:58.454606 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.454554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.455215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.455049 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 13:30:58.457583 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.457562 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.457742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.457604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.461940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.461829 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.462050 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.462022 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 13:30:58.462108 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.462064 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.462351 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.462324 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w88gf\"" Sep 10 13:30:58.465470 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.465451 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k67vc"] Sep 10 13:30:58.465624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.465596 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.468072 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.468056 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.468345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.468326 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 13:30:58.468712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.468614 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 13:30:58.469225 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.469197 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cwqkt\"" Sep 10 13:30:58.469309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.469244 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.472040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.471987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.474077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.473996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swtn\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.480639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480719 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-49wvj"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480785 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dkbfp"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgjcm"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480943 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480963 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccxnk"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.480997 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k67vc"] Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.481016 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 13:30:58.481794 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.481102 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" Sep 10 13:30:58.484530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.484418 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 13:30:58.485248 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.485225 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 13:30:58.485588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.485568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9khhz\"" Sep 10 13:30:58.488555 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.488538 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n9twt"] Sep 10 13:30:58.489909 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.489888 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-676fd4f68-q9ztc"] Sep 10 13:30:58.552751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxzq\" (UniqueName: \"kubernetes.io/projected/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-kube-api-access-7wxzq\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.552751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-stats-auth\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9936248a-cfed-420b-914f-b5ebc63997ea-tmp-dir\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-tmp\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.552998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.552994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-default-certificate\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.553271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-config\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.553271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea039094-07d1-4d6b-98b1-04f6134ed357-serving-cert\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.553271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxxw\" (UniqueName: \"kubernetes.io/projected/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-kube-api-access-9zxxw\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.553271 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9936248a-cfed-420b-914f-b5ebc63997ea-tmp-dir\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv76l\" (UniqueName: \"kubernetes.io/projected/7821b6a2-3699-4673-a5ef-b69541733215-kube-api-access-rv76l\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kzp\" (UniqueName: \"kubernetes.io/projected/97c642c5-28bd-4ea9-97dd-311e2d943a9a-kube-api-access-v5kzp\") pod \"volume-data-source-validator-695c895bd6-l5l4s\" (UID: \"97c642c5-28bd-4ea9-97dd-311e2d943a9a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gst2k\" (UniqueName: \"kubernetes.io/projected/0dbe2dc5-3878-4994-be23-bcedf15db377-kube-api-access-gst2k\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e916d2c-1680-46eb-8877-f7b20ef70b81-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553454 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:30:58.553489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e916d2c-1680-46eb-8877-f7b20ef70b81-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-serving-cert\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553521 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.053497952 +0000 UTC m=+34.663282863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553546 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-trusted-ca\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553591 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.053579377 +0000 UTC m=+34.663364282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-snapshots\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fr7\" (UniqueName: \"kubernetes.io/projected/9936248a-cfed-420b-914f-b5ebc63997ea-kube-api-access-q9fr7\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76nm\" (UniqueName: \"kubernetes.io/projected/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-kube-api-access-s76nm\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fldp9\" (UniqueName: \"kubernetes.io/projected/d373a59e-ec7a-4648-b392-e750f5f3d554-kube-api-access-fldp9\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-nginx-conf\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.553878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-serving-cert\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea039094-07d1-4d6b-98b1-04f6134ed357-config\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553926 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553953 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.553961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45s6g\" (UniqueName: \"kubernetes.io/projected/3e916d2c-1680-46eb-8877-f7b20ef70b81-kube-api-access-45s6g\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.553998 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.053957821 +0000 UTC m=+34.663742717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.554021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret podName:7954047b-1d2d-4a68-8df2-db357abdf494 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.554011869 +0000 UTC m=+66.163796774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret") pod "global-pull-secret-syncer-kh6q9" (UID: "7954047b-1d2d-4a68-8df2-db357abdf494") : object "kube-system"/"original-pull-secret" not registered Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554057 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d373a59e-ec7a-4648-b392-e750f5f3d554-cert\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwcs\" (UniqueName: \"kubernetes.io/projected/ea039094-07d1-4d6b-98b1-04f6134ed357-kube-api-access-dzwcs\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9936248a-cfed-420b-914f-b5ebc63997ea-config-volume\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea039094-07d1-4d6b-98b1-04f6134ed357-config\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-nginx-conf\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554715 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.555012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.554763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9936248a-cfed-420b-914f-b5ebc63997ea-config-volume\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.556039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.555888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea039094-07d1-4d6b-98b1-04f6134ed357-serving-cert\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.571531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.571476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwcs\" (UniqueName: \"kubernetes.io/projected/ea039094-07d1-4d6b-98b1-04f6134ed357-kube-api-access-dzwcs\") pod \"service-ca-operator-859f94c4b-j8spv\" (UID: \"ea039094-07d1-4d6b-98b1-04f6134ed357\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.571821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.571797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxzq\" (UniqueName: \"kubernetes.io/projected/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-kube-api-access-7wxzq\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:58.572126 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.572053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fr7\" (UniqueName: \"kubernetes.io/projected/9936248a-cfed-420b-914f-b5ebc63997ea-kube-api-access-q9fr7\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:58.573708 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.573687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kzp\" (UniqueName: \"kubernetes.io/projected/97c642c5-28bd-4ea9-97dd-311e2d943a9a-kube-api-access-v5kzp\") pod \"volume-data-source-validator-695c895bd6-l5l4s\" (UID: \"97c642c5-28bd-4ea9-97dd-311e2d943a9a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" Sep 10 13:30:58.591185 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.591150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:30:58.614143 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.614121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" Sep 10 13:30:58.654795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-snapshots\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.654906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.654906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:58.654906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76nm\" (UniqueName: \"kubernetes.io/projected/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-kube-api-access-s76nm\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.654906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fldp9\" (UniqueName: \"kubernetes.io/projected/d373a59e-ec7a-4648-b392-e750f5f3d554-kube-api-access-fldp9\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.655099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-serving-cert\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.655099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.654998 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45s6g\" (UniqueName: \"kubernetes.io/projected/3e916d2c-1680-46eb-8877-f7b20ef70b81-kube-api-access-45s6g\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.655099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.655099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d373a59e-ec7a-4648-b392-e750f5f3d554-cert\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.655099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-stats-auth\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-tmp\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-default-certificate\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-config\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxxw\" (UniqueName: \"kubernetes.io/projected/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-kube-api-access-9zxxw\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv76l\" (UniqueName: \"kubernetes.io/projected/7821b6a2-3699-4673-a5ef-b69541733215-kube-api-access-rv76l\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-snapshots\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mrt\" (UniqueName: \"kubernetes.io/projected/7025aadd-365d-498c-8792-b068ddff934b-kube-api-access-x6mrt\") pod \"network-check-source-685456655f-k67vc\" (UID: \"7025aadd-365d-498c-8792-b068ddff934b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gst2k\" (UniqueName: \"kubernetes.io/projected/0dbe2dc5-3878-4994-be23-bcedf15db377-kube-api-access-gst2k\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e916d2c-1680-46eb-8877-f7b20ef70b81-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e916d2c-1680-46eb-8877-f7b20ef70b81-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.655524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-serving-cert\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.656160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.655549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-trusted-ca\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.656160 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.655799 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:58.656160 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.655864 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs podName:22a2afc9-2500-44a2-9f17-b886c4ec7142 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.655846601 +0000 UTC m=+66.265631509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs") pod "network-metrics-daemon-c5z5r" (UID: "22a2afc9-2500-44a2-9f17-b886c4ec7142") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 13:30:58.656160 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.656132 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:30:58.656402 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.656188 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.156170512 +0000 UTC m=+34.765955408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:30:58.656472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.656451 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-trusted-ca\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.656577 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.656562 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.156548276 +0000 UTC m=+34.766333186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:30:58.656652 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.656597 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:30:58.656652 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.656643 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.156626755 +0000 UTC m=+34.766411675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:30:58.657084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.656988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.657165 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.657133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-tmp\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.657389 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.657339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e916d2c-1680-46eb-8877-f7b20ef70b81-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.657493 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.657465 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-config\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.657563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.657523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.658233 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.658207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-stats-auth\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.659648 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.659624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e916d2c-1680-46eb-8877-f7b20ef70b81-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.659884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.659864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-serving-cert\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.659884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.659872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-default-certificate\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.661267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.661161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d373a59e-ec7a-4648-b392-e750f5f3d554-cert\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.662164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.662141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-serving-cert\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.667512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.667471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fldp9\" (UniqueName: \"kubernetes.io/projected/d373a59e-ec7a-4648-b392-e750f5f3d554-kube-api-access-fldp9\") pod \"ingress-canary-n9twt\" (UID: \"d373a59e-ec7a-4648-b392-e750f5f3d554\") " pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.667757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.667739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxxw\" (UniqueName: \"kubernetes.io/projected/6aeb68d5-6782-45d0-8aff-fa15cd8e9266-kube-api-access-9zxxw\") pod \"console-operator-85b454874b-zgjcm\" (UID: \"6aeb68d5-6782-45d0-8aff-fa15cd8e9266\") " pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.668711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.668670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv76l\" (UniqueName: \"kubernetes.io/projected/7821b6a2-3699-4673-a5ef-b69541733215-kube-api-access-rv76l\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:58.670963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.670930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gst2k\" (UniqueName: \"kubernetes.io/projected/0dbe2dc5-3878-4994-be23-bcedf15db377-kube-api-access-gst2k\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:58.671196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.671171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45s6g\" (UniqueName: \"kubernetes.io/projected/3e916d2c-1680-46eb-8877-f7b20ef70b81-kube-api-access-45s6g\") pod \"kube-storage-version-migrator-operator-8d7dd4675-kt5jf\" (UID: \"3e916d2c-1680-46eb-8877-f7b20ef70b81\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.671268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.671177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76nm\" (UniqueName: \"kubernetes.io/projected/2fd2d410-6c72-4e23-8d17-d1554cbe69e8-kube-api-access-s76nm\") pod \"insights-operator-76bcd6ccf5-dkbfp\" (UID: \"2fd2d410-6c72-4e23-8d17-d1554cbe69e8\") " pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.677259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.677242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" Sep 10 13:30:58.692047 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.691996 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" Sep 10 13:30:58.710430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.710409 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:30:58.757563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.756721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mrt\" (UniqueName: \"kubernetes.io/projected/7025aadd-365d-498c-8792-b068ddff934b-kube-api-access-x6mrt\") pod \"network-check-source-685456655f-k67vc\" (UID: \"7025aadd-365d-498c-8792-b068ddff934b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" Sep 10 13:30:58.757563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.756938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:58.761310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.761271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2hk\" (UniqueName: \"kubernetes.io/projected/02dc4a55-e539-4eea-960e-46fb50b74ad5-kube-api-access-sg2hk\") pod \"network-check-target-9htdb\" (UID: \"02dc4a55-e539-4eea-960e-46fb50b74ad5\") " pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:58.773390 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.773294 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n9twt" Sep 10 13:30:58.778285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.778219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mrt\" (UniqueName: \"kubernetes.io/projected/7025aadd-365d-498c-8792-b068ddff934b-kube-api-access-x6mrt\") pod \"network-check-source-685456655f-k67vc\" (UID: \"7025aadd-365d-498c-8792-b068ddff934b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" Sep 10 13:30:58.783899 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.783032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:30:58.793771 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.793752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" Sep 10 13:30:58.800786 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.800572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" Sep 10 13:30:58.912594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.912422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:30:58.912594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.912435 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:30:58.912594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.912579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:58.915531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.915308 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twwrf\"" Sep 10 13:30:58.915531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.915440 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:30:58.915531 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.915491 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nc4jc\"" Sep 10 13:30:58.915744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.915545 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:30:58.938202 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.938184 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:30:58.956049 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.955999 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv"] Sep 10 13:30:58.958802 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:58.958707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:58.958802 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:58.958725 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea039094_07d1_4d6b_98b1_04f6134ed357.slice/crio-5cb54d1fe0398c2fef28eb4dd1f5e589ca93eb975068b36c6f2f6af4f068dbfc WatchSource:0}: Error finding container 5cb54d1fe0398c2fef28eb4dd1f5e589ca93eb975068b36c6f2f6af4f068dbfc: Status 404 returned error can't find the container with id 5cb54d1fe0398c2fef28eb4dd1f5e589ca93eb975068b36c6f2f6af4f068dbfc Sep 10 13:30:58.958927 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.958842 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:30:58.958927 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.958860 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:30:58.958927 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:58.958921 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.958903201 +0000 UTC m=+35.568688099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:30:59.009471 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.009143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s"] Sep 10 13:30:59.012223 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.012193 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c642c5_28bd_4ea9_97dd_311e2d943a9a.slice/crio-2177d6467fb086fd5b011ca7d2fe79a89f6e23b5f49d851623ad9b63455dc440 WatchSource:0}: Error finding container 2177d6467fb086fd5b011ca7d2fe79a89f6e23b5f49d851623ad9b63455dc440: Status 404 returned error can't find the container with id 2177d6467fb086fd5b011ca7d2fe79a89f6e23b5f49d851623ad9b63455dc440 Sep 10 13:30:59.040071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.040043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf"] Sep 10 13:30:59.042482 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.042455 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e916d2c_1680_46eb_8877_f7b20ef70b81.slice/crio-1469d52f6bd7c47711d760d42769c757d7a9d97ca2bf335110dc44f8282d9375 WatchSource:0}: Error finding container 1469d52f6bd7c47711d760d42769c757d7a9d97ca2bf335110dc44f8282d9375: Status 404 returned error can't find the container with id 1469d52f6bd7c47711d760d42769c757d7a9d97ca2bf335110dc44f8282d9375 Sep 10 13:30:59.054687 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.054647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgjcm"] Sep 10 13:30:59.056929 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.056905 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb68d5_6782_45d0_8aff_fa15cd8e9266.slice/crio-ed0a54e12d5cb7d4ca543b8ddcc51b58cd865ecf0f40935c483ce802a6714afc WatchSource:0}: Error finding container ed0a54e12d5cb7d4ca543b8ddcc51b58cd865ecf0f40935c483ce802a6714afc: Status 404 returned error can't find the container with id ed0a54e12d5cb7d4ca543b8ddcc51b58cd865ecf0f40935c483ce802a6714afc Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.059139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.059174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.059268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059489 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059543 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.059522727 +0000 UTC m=+35.669307631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059611 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059645 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.059633286 +0000 UTC m=+35.669418198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059704 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:30:59.059748 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.059735 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.05972588 +0000 UTC m=+35.669510801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:30:59.070277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.070250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" event={"ID":"6aeb68d5-6782-45d0-8aff-fa15cd8e9266","Type":"ContainerStarted","Data":"ed0a54e12d5cb7d4ca543b8ddcc51b58cd865ecf0f40935c483ce802a6714afc"} Sep 10 13:30:59.071972 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.071937 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" event={"ID":"97c642c5-28bd-4ea9-97dd-311e2d943a9a","Type":"ContainerStarted","Data":"2177d6467fb086fd5b011ca7d2fe79a89f6e23b5f49d851623ad9b63455dc440"} Sep 10 13:30:59.073299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.073273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" event={"ID":"ea039094-07d1-4d6b-98b1-04f6134ed357","Type":"ContainerStarted","Data":"5cb54d1fe0398c2fef28eb4dd1f5e589ca93eb975068b36c6f2f6af4f068dbfc"} Sep 10 13:30:59.074610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.074583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" event={"ID":"3e916d2c-1680-46eb-8877-f7b20ef70b81","Type":"ContainerStarted","Data":"1469d52f6bd7c47711d760d42769c757d7a9d97ca2bf335110dc44f8282d9375"} Sep 10 13:30:59.087461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.087434 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n9twt"] Sep 10 13:30:59.092018 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.091938 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd373a59e_ec7a_4648_b392_e750f5f3d554.slice/crio-60e5cbc413bd8bf5c2bb335f6950b33507d8fe4564a129806a3742f027b94668 WatchSource:0}: Error finding container 60e5cbc413bd8bf5c2bb335f6950b33507d8fe4564a129806a3742f027b94668: Status 404 returned error can't find the container with id 60e5cbc413bd8bf5c2bb335f6950b33507d8fe4564a129806a3742f027b94668 Sep 10 13:30:59.138695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.138664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-dkbfp"] Sep 10 13:30:59.141903 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.141865 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd2d410_6c72_4e23_8d17_d1554cbe69e8.slice/crio-544dcfaa761ae890e027d8ce094d2688f5f84f559df92da5ebfbd41d559895d4 WatchSource:0}: Error finding container 544dcfaa761ae890e027d8ce094d2688f5f84f559df92da5ebfbd41d559895d4: Status 404 returned error can't find the container with id 544dcfaa761ae890e027d8ce094d2688f5f84f559df92da5ebfbd41d559895d4 Sep 10 13:30:59.143808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.143763 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-k67vc"] Sep 10 13:30:59.146125 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.146064 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7025aadd_365d_498c_8792_b068ddff934b.slice/crio-5566725911cae1e5ee0e29d76c677af9ae31a2825f6f4375def938ec933d8012 WatchSource:0}: Error finding container 5566725911cae1e5ee0e29d76c677af9ae31a2825f6f4375def938ec933d8012: Status 404 returned error can't find the container with id 5566725911cae1e5ee0e29d76c677af9ae31a2825f6f4375def938ec933d8012 Sep 10 13:30:59.160037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.160013 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:30:59.160125 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.160078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:59.160125 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.160106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:30:59.160198 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.160151 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:30:59.160244 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.160200 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:30:59.160244 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.160211 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.160194251 +0000 UTC m=+35.769979142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:30:59.160244 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.160230 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.160220695 +0000 UTC m=+35.770005588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:30:59.160461 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.160245 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:00.160237609 +0000 UTC m=+35.770022501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:30:59.248359 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.248281 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9htdb"] Sep 10 13:30:59.251284 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:30:59.251258 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dc4a55_e539_4eea_960e_46fb50b74ad5.slice/crio-728e32e325ccb7d99b1c8681c53f2295f1b3a35565e518df84239d513b7f37d2 WatchSource:0}: Error finding container 728e32e325ccb7d99b1c8681c53f2295f1b3a35565e518df84239d513b7f37d2: Status 404 returned error can't find the container with id 728e32e325ccb7d99b1c8681c53f2295f1b3a35565e518df84239d513b7f37d2 Sep 10 13:30:59.968066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:30:59.967491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:30:59.968066 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.967627 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:30:59.968066 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.967649 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:30:59.968066 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:30:59.967718 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:01.967695775 +0000 UTC m=+37.577480682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:31:00.068558 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.068525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:00.068722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.068569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:00.068722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.068668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:00.068856 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.068822 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:00.068904 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.068896 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.06887576 +0000 UTC m=+37.678660657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:00.069310 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.069287 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:31:00.069416 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.069353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.069336024 +0000 UTC m=+37.679120934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:31:00.069481 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.069428 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:31:00.069481 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.069459 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.069448645 +0000 UTC m=+37.679233551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:31:00.081578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.081546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" event={"ID":"2fd2d410-6c72-4e23-8d17-d1554cbe69e8","Type":"ContainerStarted","Data":"544dcfaa761ae890e027d8ce094d2688f5f84f559df92da5ebfbd41d559895d4"} Sep 10 13:31:00.084450 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.084424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9htdb" event={"ID":"02dc4a55-e539-4eea-960e-46fb50b74ad5","Type":"ContainerStarted","Data":"728e32e325ccb7d99b1c8681c53f2295f1b3a35565e518df84239d513b7f37d2"} Sep 10 13:31:00.086438 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.086412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n9twt" event={"ID":"d373a59e-ec7a-4648-b392-e750f5f3d554","Type":"ContainerStarted","Data":"60e5cbc413bd8bf5c2bb335f6950b33507d8fe4564a129806a3742f027b94668"} Sep 10 13:31:00.090550 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.090388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" event={"ID":"7025aadd-365d-498c-8792-b068ddff934b","Type":"ContainerStarted","Data":"5566725911cae1e5ee0e29d76c677af9ae31a2825f6f4375def938ec933d8012"} Sep 10 13:31:00.170391 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.169659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:00.170391 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.169735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:00.170391 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:00.169818 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:00.170391 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.169998 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:31:00.170391 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.170058 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.170040664 +0000 UTC m=+37.779825578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:31:00.170753 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.170457 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.17044198 +0000 UTC m=+37.780226871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:31:00.170753 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.170524 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:31:00.170753 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:00.170557 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:02.170545936 +0000 UTC m=+37.780330843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:31:01.988517 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:01.988479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:01.988983 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:01.988718 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:31:01.988983 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:01.988734 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:31:01.988983 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:01.988793 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:05.988773839 +0000 UTC m=+41.598558743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:31:02.089129 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.089094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:02.089292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.089235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:02.089292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.089266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:02.089463 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.089445 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:31:02.089585 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.089510 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.089492373 +0000 UTC m=+41.699277268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:31:02.089916 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.089896 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:02.089986 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.089958 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.089941861 +0000 UTC m=+41.699726758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:02.090049 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.090040 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:31:02.090098 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.090074 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.090063429 +0000 UTC m=+41.699848324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:31:02.190055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.190020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:02.190223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.190124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:02.190288 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:02.190218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:02.190739 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.190424 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.190401635 +0000 UTC m=+41.800186561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:31:02.190876 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.190788 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:31:02.190876 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.190845 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.190832382 +0000 UTC m=+41.800617276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:31:02.190876 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.190790 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:31:02.190876 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:02.190878 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:06.190872336 +0000 UTC m=+41.800657230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:31:03.001709 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:03.001606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:03.197241 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:03.197199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:04.007396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:04.007329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:04.289711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:04.289634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:04.552144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:04.552065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:06.020132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.020097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:06.020613 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.020262 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:31:06.020613 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.020280 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:31:06.020613 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.020346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.020323633 +0000 UTC m=+49.630108526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:31:06.121298 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.121259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:06.121456 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.121379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:06.121456 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.121403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:06.121456 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121441 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:06.121597 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121503 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:31:06.121597 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121503 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:31:06.121597 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121508 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.121492938 +0000 UTC m=+49.731277828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:06.121597 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121584 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.121565481 +0000 UTC m=+49.731350385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:31:06.121760 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.121603 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.121596205 +0000 UTC m=+49.731381099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:31:06.222578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.222546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:06.222757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.222630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:06.222757 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.222697 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:31:06.222757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:06.222712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:06.222757 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.222750 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.222736077 +0000 UTC m=+49.832520973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:31:06.222966 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.222840 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.222823781 +0000 UTC m=+49.832608692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:31:06.222966 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.222918 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:31:06.222966 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:06.222957 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.222945447 +0000 UTC m=+49.832730343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:31:09.120497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:09.120467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" event={"ID":"97c642c5-28bd-4ea9-97dd-311e2d943a9a","Type":"ContainerStarted","Data":"64cab74cf4da52b4253a11f4d8cac8dcbc854e827de026ef02036cc0427444a3"} Sep 10 13:31:09.813224 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:09.813169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:09.934862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:09.934833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:10.126861 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.126444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" event={"ID":"7025aadd-365d-498c-8792-b068ddff934b","Type":"ContainerStarted","Data":"02230068a28d755336c06634bd44abaa72f073362d5b986140b498806aef1789"} Sep 10 13:31:10.129528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.129504 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="d37bc5bb683ae2612d73630367ed3615fede8709287fabe171f28cb3bb637852" exitCode=0 Sep 10 13:31:10.129632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.129576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"d37bc5bb683ae2612d73630367ed3615fede8709287fabe171f28cb3bb637852"} Sep 10 13:31:10.131546 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.131479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" event={"ID":"2fd2d410-6c72-4e23-8d17-d1554cbe69e8","Type":"ContainerStarted","Data":"e226b4d21bc17623e79d399b6827c6cca589960a154dbce9ef10b360d91c3871"} Sep 10 13:31:10.133659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.133638 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/0.log" Sep 10 13:31:10.133758 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.133678 2559 generic.go:356] "Generic (PLEG): container finished" podID="6aeb68d5-6782-45d0-8aff-fa15cd8e9266" containerID="7f48e0025669d42876521fa203a6d0f5ab1359809980001efd187690420c3468" exitCode=255 Sep 10 13:31:10.133815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.133763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" event={"ID":"6aeb68d5-6782-45d0-8aff-fa15cd8e9266","Type":"ContainerDied","Data":"7f48e0025669d42876521fa203a6d0f5ab1359809980001efd187690420c3468"} Sep 10 13:31:10.133933 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.133916 2559 scope.go:117] "RemoveContainer" containerID="7f48e0025669d42876521fa203a6d0f5ab1359809980001efd187690420c3468" Sep 10 13:31:10.136976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.136452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" event={"ID":"ea039094-07d1-4d6b-98b1-04f6134ed357","Type":"ContainerStarted","Data":"b1ac8bd903c23bf4e34f104235771b9df50d4bb73e2170f6e3314c241121ae0c"} Sep 10 13:31:10.138418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.138394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9htdb" event={"ID":"02dc4a55-e539-4eea-960e-46fb50b74ad5","Type":"ContainerStarted","Data":"45343703c83a42431daca552c9974cde60500a0741eabfe610ecc8e954abb2dc"} Sep 10 13:31:10.140142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.140113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n9twt" event={"ID":"d373a59e-ec7a-4648-b392-e750f5f3d554","Type":"ContainerStarted","Data":"b2162e8e8272b8fa020b6d61196252bdd5a058cc36849bf1f1cd7bf5474ee0f4"} Sep 10 13:31:10.141884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.141862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" event={"ID":"3e916d2c-1680-46eb-8877-f7b20ef70b81","Type":"ContainerStarted","Data":"54593538e75af071a1970486af26097f04d90343ef96a7c4f886c71cb21f0b02"} Sep 10 13:31:10.154064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.154021 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-k67vc" podStartSLOduration=15.313953208 podStartE2EDuration="25.154005605s" totalImagesPullingTime="9.840052397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.152956384 +0000 UTC m=+45.762741298" watchObservedRunningTime="2026-09-10 13:31:10.154005605 +0000 UTC m=+45.763790520" Sep 10 13:31:10.167925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.167869 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:31:10.178749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.178706 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" podStartSLOduration=15.32888695 podStartE2EDuration="25.178688587s" totalImagesPullingTime="9.849801637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.177600831 +0000 UTC m=+45.787385736" watchObservedRunningTime="2026-09-10 13:31:10.178688587 +0000 UTC m=+45.788473501" Sep 10 13:31:10.309266 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.309204 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" podStartSLOduration=15.28381838 podStartE2EDuration="25.309183966s" totalImagesPullingTime="10.025365586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.307427351 +0000 UTC m=+45.917212257" watchObservedRunningTime="2026-09-10 13:31:10.309183966 +0000 UTC m=+45.918968880" Sep 10 13:31:10.353252 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.352192 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n9twt" podStartSLOduration=2.475603057 podStartE2EDuration="12.352171781s" totalImagesPullingTime="9.876568724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.351395389 +0000 UTC m=+45.961180304" watchObservedRunningTime="2026-09-10 13:31:10.352171781 +0000 UTC m=+45.961956695" Sep 10 13:31:10.372761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.372092 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-l5l4s" podStartSLOduration=16.216284305 podStartE2EDuration="25.372073469s" totalImagesPullingTime="9.155789164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.371480487 +0000 UTC m=+45.981265401" watchObservedRunningTime="2026-09-10 13:31:10.372073469 +0000 UTC m=+45.981858381" Sep 10 13:31:10.394784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.394594 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" podStartSLOduration=15.451190214 podStartE2EDuration="25.394574542s" totalImagesPullingTime="9.943384328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.39322967 +0000 UTC m=+46.003014579" watchObservedRunningTime="2026-09-10 13:31:10.394574542 +0000 UTC m=+46.004359456" Sep 10 13:31:10.416222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.416179 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9htdb" podStartSLOduration=35.69758408 podStartE2EDuration="45.416166751s" totalImagesPullingTime="9.718582671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:10.415202266 +0000 UTC m=+46.024987179" watchObservedRunningTime="2026-09-10 13:31:10.416166751 +0000 UTC m=+46.025951665" Sep 10 13:31:10.603540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.603510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:10.898495 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.897856 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8"] Sep 10 13:31:10.919704 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.919662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8"] Sep 10 13:31:10.920009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.919989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" Sep 10 13:31:10.923144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.922522 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 13:31:10.923144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.922794 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 13:31:10.923666 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.923635 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vz9sz\"" Sep 10 13:31:10.964931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:10.964905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmkh\" (UniqueName: \"kubernetes.io/projected/ff92bed8-0c9a-4969-bbe8-9e322591cbc2-kube-api-access-qpmkh\") pod \"migrator-fdbb8fdff-z2bk8\" (UID: \"ff92bed8-0c9a-4969-bbe8-9e322591cbc2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" Sep 10 13:31:11.065439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.065406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmkh\" (UniqueName: \"kubernetes.io/projected/ff92bed8-0c9a-4969-bbe8-9e322591cbc2-kube-api-access-qpmkh\") pod \"migrator-fdbb8fdff-z2bk8\" (UID: \"ff92bed8-0c9a-4969-bbe8-9e322591cbc2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" Sep 10 13:31:11.070778 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.070754 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v9wl8"] Sep 10 13:31:11.086920 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.086893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmkh\" (UniqueName: \"kubernetes.io/projected/ff92bed8-0c9a-4969-bbe8-9e322591cbc2-kube-api-access-qpmkh\") pod \"migrator-fdbb8fdff-z2bk8\" (UID: \"ff92bed8-0c9a-4969-bbe8-9e322591cbc2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" Sep 10 13:31:11.092940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.092923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.095709 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.095677 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2d75h\"" Sep 10 13:31:11.096238 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.096044 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 13:31:11.096545 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.096523 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 13:31:11.097611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.097594 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9wl8"] Sep 10 13:31:11.147928 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.147900 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd7ad402-4d5f-4cd8-961d-61286e9a9678" containerID="9eea5f4571a3cbaa4beff73893fd661d123b49b5378c457af3477978043770ea" exitCode=0 Sep 10 13:31:11.148236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.147982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerDied","Data":"9eea5f4571a3cbaa4beff73893fd661d123b49b5378c457af3477978043770ea"} Sep 10 13:31:11.149594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.149578 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:31:11.150024 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.150010 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/0.log" Sep 10 13:31:11.150093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.150041 2559 generic.go:356] "Generic (PLEG): container finished" podID="6aeb68d5-6782-45d0-8aff-fa15cd8e9266" containerID="080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094" exitCode=255 Sep 10 13:31:11.150156 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.150123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" event={"ID":"6aeb68d5-6782-45d0-8aff-fa15cd8e9266","Type":"ContainerDied","Data":"080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094"} Sep 10 13:31:11.150209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.150177 2559 scope.go:117] "RemoveContainer" containerID="7f48e0025669d42876521fa203a6d0f5ab1359809980001efd187690420c3468" Sep 10 13:31:11.150396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.150356 2559 scope.go:117] "RemoveContainer" containerID="080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094" Sep 10 13:31:11.150555 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.150536 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgjcm_openshift-console-operator(6aeb68d5-6782-45d0-8aff-fa15cd8e9266)\"" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" podUID="6aeb68d5-6782-45d0-8aff-fa15cd8e9266" Sep 10 13:31:11.166749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.166705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.166971 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.166947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-data-volume\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.167051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.166993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h2t\" (UniqueName: \"kubernetes.io/projected/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-api-access-q7h2t\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.167201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.167175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.167295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.167234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-crio-socket\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.232413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.232380 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" Sep 10 13:31:11.268085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-data-volume\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h2t\" (UniqueName: \"kubernetes.io/projected/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-api-access-q7h2t\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268161 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268186 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-crio-socket\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268285 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.268250 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:31:11.268407 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.268320 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:11.768301492 +0000 UTC m=+47.378086399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : configmap "kube-rbac-proxy" not found Sep 10 13:31:11.268407 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-crio-socket\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.268429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-data-volume\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.268614 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.268535 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:31:11.268614 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.268587 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:11.768569393 +0000 UTC m=+47.378354305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : secret "insights-runtime-extractor-tls" not found Sep 10 13:31:11.285578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.285551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h2t\" (UniqueName: \"kubernetes.io/projected/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-api-access-q7h2t\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.542320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.542300 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8"] Sep 10 13:31:11.544736 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:11.544715 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff92bed8_0c9a_4969_bbe8_9e322591cbc2.slice/crio-94bd5f4f83de9dc85a4d861a73bad2d8c33be54f2f6bb4f8b33eb7fe80ed6174 WatchSource:0}: Error finding container 94bd5f4f83de9dc85a4d861a73bad2d8c33be54f2f6bb4f8b33eb7fe80ed6174: Status 404 returned error can't find the container with id 94bd5f4f83de9dc85a4d861a73bad2d8c33be54f2f6bb4f8b33eb7fe80ed6174 Sep 10 13:31:11.771762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.771735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.771851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.771811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:11.771892 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.771848 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:31:11.771926 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.771914 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:12.771898093 +0000 UTC m=+48.381683005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : configmap "kube-rbac-proxy" not found Sep 10 13:31:11.771926 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.771915 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:31:11.771997 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:11.771954 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:12.771943321 +0000 UTC m=+48.381728212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : secret "insights-runtime-extractor-tls" not found Sep 10 13:31:11.803693 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.803641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:11.902353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:11.902331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:12.155089 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.155036 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" event={"ID":"ff92bed8-0c9a-4969-bbe8-9e322591cbc2","Type":"ContainerStarted","Data":"94bd5f4f83de9dc85a4d861a73bad2d8c33be54f2f6bb4f8b33eb7fe80ed6174"} Sep 10 13:31:12.158318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.158296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8x94" event={"ID":"fd7ad402-4d5f-4cd8-961d-61286e9a9678","Type":"ContainerStarted","Data":"5f3dd0495c9500e001ecac02eccdf154b2ff68b7533153e5ab895498dcd59b20"} Sep 10 13:31:12.159770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.159752 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:31:12.160015 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.160001 2559 scope.go:117] "RemoveContainer" containerID="080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094" Sep 10 13:31:12.160145 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:12.160128 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgjcm_openshift-console-operator(6aeb68d5-6782-45d0-8aff-fa15cd8e9266)\"" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" podUID="6aeb68d5-6782-45d0-8aff-fa15cd8e9266" Sep 10 13:31:12.228580 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.228532 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8x94" podStartSLOduration=7.52294195 podStartE2EDuration="47.228515424s" totalImagesPullingTime="39.705573474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:12.194628172 +0000 UTC m=+47.804413097" watchObservedRunningTime="2026-09-10 13:31:12.228515424 +0000 UTC m=+47.838300336" Sep 10 13:31:12.390014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.389978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:12.520160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.520130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:12.781601 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.781537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:12.781732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.781638 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:12.781732 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:12.781663 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 13:31:12.781732 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:12.781724 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.781706246 +0000 UTC m=+50.391491157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : configmap "kube-rbac-proxy" not found Sep 10 13:31:12.781905 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:12.781809 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:31:12.781905 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:12.781892 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:14.781873636 +0000 UTC m=+50.391658534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : secret "insights-runtime-extractor-tls" not found Sep 10 13:31:12.860822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.860791 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d246c"] Sep 10 13:31:12.883530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.883496 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d246c"] Sep 10 13:31:12.883679 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.883637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:12.887310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.887279 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 13:31:12.887911 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.887764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 13:31:12.889429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.889225 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 13:31:12.891081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.889788 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 13:31:12.891081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.890016 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kb799\"" Sep 10 13:31:12.972012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.971980 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:12.983047 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.983020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trr7p\" (UniqueName: \"kubernetes.io/projected/2e2853e4-2cec-4645-a7d0-790a14232454-kube-api-access-trr7p\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:12.983153 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.983141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e2853e4-2cec-4645-a7d0-790a14232454-signing-cabundle\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:12.983232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:12.983209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e2853e4-2cec-4645-a7d0-790a14232454-signing-key\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.061014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.060960 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:13.084814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.084600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e2853e4-2cec-4645-a7d0-790a14232454-signing-cabundle\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.084814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.084656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e2853e4-2cec-4645-a7d0-790a14232454-signing-key\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.084814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.084734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trr7p\" (UniqueName: \"kubernetes.io/projected/2e2853e4-2cec-4645-a7d0-790a14232454-kube-api-access-trr7p\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.085707 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.085683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2e2853e4-2cec-4645-a7d0-790a14232454-signing-cabundle\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.088902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.088880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2e2853e4-2cec-4645-a7d0-790a14232454-signing-key\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.098616 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.098592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trr7p\" (UniqueName: \"kubernetes.io/projected/2e2853e4-2cec-4645-a7d0-790a14232454-kube-api-access-trr7p\") pod \"service-ca-75d99c484-d246c\" (UID: \"2e2853e4-2cec-4645-a7d0-790a14232454\") " pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.195726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.195708 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wzz2s_5828c67b-c93e-4af9-946e-2da852db81e3/dns-node-resolver/0.log" Sep 10 13:31:13.196793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.196777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d246c" Sep 10 13:31:13.314884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.314826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:13.386616 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.386589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:13.686686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:13.686666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d246c"] Sep 10 13:31:13.688774 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:13.688742 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2853e4_2cec_4645_a7d0_790a14232454.slice/crio-6aa1aac3b0dc4b147edd3271e03a7af737235f121c451dbc9231c68fe706657b WatchSource:0}: Error finding container 6aa1aac3b0dc4b147edd3271e03a7af737235f121c451dbc9231c68fe706657b: Status 404 returned error can't find the container with id 6aa1aac3b0dc4b147edd3271e03a7af737235f121c451dbc9231c68fe706657b Sep 10 13:31:14.092849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.092814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:14.093007 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.092954 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 13:31:14.093007 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.092973 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d66d65f69-rsshw: secret "image-registry-tls" not found Sep 10 13:31:14.093121 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.093026 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls podName:4a86429b-d86d-4871-984b-f4ee5212f950 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.093012534 +0000 UTC m=+65.702797443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls") pod "image-registry-5d66d65f69-rsshw" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950") : secret "image-registry-tls" not found Sep 10 13:31:14.168543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.168509 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d246c" event={"ID":"2e2853e4-2cec-4645-a7d0-790a14232454","Type":"ContainerStarted","Data":"9eef8ff6bdf98fd9d0c7a022c5361fef64a2b228e3b9b9ac3d6746b37e88aa0d"} Sep 10 13:31:14.168675 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.168564 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d246c" event={"ID":"2e2853e4-2cec-4645-a7d0-790a14232454","Type":"ContainerStarted","Data":"6aa1aac3b0dc4b147edd3271e03a7af737235f121c451dbc9231c68fe706657b"} Sep 10 13:31:14.170297 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.170275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" event={"ID":"ff92bed8-0c9a-4969-bbe8-9e322591cbc2","Type":"ContainerStarted","Data":"1277a230c4feb5143decdd0d30374b101f1138d4c4869ecd093e4020d58c77e9"} Sep 10 13:31:14.170420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.170322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" event={"ID":"ff92bed8-0c9a-4969-bbe8-9e322591cbc2","Type":"ContainerStarted","Data":"f54faebe23b7fd2dbce5e0e3238bd8300b024b1b943ea659ff530681d44b4f20"} Sep 10 13:31:14.188713 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.188664 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-d246c" podStartSLOduration=2.188644557 podStartE2EDuration="2.188644557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:14.186755372 +0000 UTC m=+49.796540288" watchObservedRunningTime="2026-09-10 13:31:14.188644557 +0000 UTC m=+49.798429471" Sep 10 13:31:14.193134 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.193105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:14.193269 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193250 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:14.193328 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193317 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls podName:d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.193297392 +0000 UTC m=+65.803082295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dvsk4" (UID: "d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9") : secret "cluster-monitoring-operator-tls" not found Sep 10 13:31:14.193426 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193328 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 13:31:14.193426 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193409 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert podName:b28758f1-b930-4ef4-8371-b2cb8c6e4ef4 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.193393695 +0000 UTC m=+65.803178609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-49wvj" (UID: "b28758f1-b930-4ef4-8371-b2cb8c6e4ef4") : secret "networking-console-plugin-cert" not found Sep 10 13:31:14.193426 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.193255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:14.193595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.193464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:14.193649 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193606 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 13:31:14.193704 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.193669 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls podName:9936248a-cfed-420b-914f-b5ebc63997ea nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.193652463 +0000 UTC m=+65.803437373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls") pod "dns-default-ccxnk" (UID: "9936248a-cfed-420b-914f-b5ebc63997ea") : secret "dns-default-metrics-tls" not found Sep 10 13:31:14.202027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.202008 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxlmv_80cbebcd-aa00-4d3a-98ec-0e86bb4275e7/node-ca/0.log" Sep 10 13:31:14.207064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.207023 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-z2bk8" podStartSLOduration=2.339165996 podStartE2EDuration="4.207006484s" totalImagesPullingTime="1.867840488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:14.20503355 +0000 UTC m=+49.814818455" watchObservedRunningTime="2026-09-10 13:31:14.207006484 +0000 UTC m=+49.816791398" Sep 10 13:31:14.294482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.294461 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:14.294562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.294530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:14.294610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.294569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:14.294646 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.294607 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 13:31:14.294683 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.294658 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls podName:7821b6a2-3699-4673-a5ef-b69541733215 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.29464216 +0000 UTC m=+65.904427065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-97mwn" (UID: "7821b6a2-3699-4673-a5ef-b69541733215") : secret "samples-operator-tls" not found Sep 10 13:31:14.294731 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.294688 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 13:31:14.294731 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.294696 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.294682619 +0000 UTC m=+65.904467529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : configmap references non-existent config key: service-ca.crt Sep 10 13:31:14.294731 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.294726 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs podName:0dbe2dc5-3878-4994-be23-bcedf15db377 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:30.2947146 +0000 UTC m=+65.904499494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs") pod "router-default-676fd4f68-q9ztc" (UID: "0dbe2dc5-3878-4994-be23-bcedf15db377") : secret "router-metrics-certs-default" not found Sep 10 13:31:14.338766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.338747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:14.445079 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.445003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:14.597730 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.597700 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n9twt_d373a59e-ec7a-4648-b392-e750f5f3d554/serve-healthcheck-canary/0.log" Sep 10 13:31:14.614010 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.613987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:14.789436 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.789402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:14.799509 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.799449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:14.799671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.799649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:14.799855 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.799837 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:31:14.799942 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:14.799907 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:18.79988975 +0000 UTC m=+54.409674666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : secret "insights-runtime-extractor-tls" not found Sep 10 13:31:14.800018 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.799973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:14.805308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.805287 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-z2bk8_ff92bed8-0c9a-4969-bbe8-9e322591cbc2/migrator/0.log" Sep 10 13:31:14.903113 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.903089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:14.996270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:14.996246 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-z2bk8_ff92bed8-0c9a-4969-bbe8-9e322591cbc2/graceful-termination/0.log" Sep 10 13:31:15.014823 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:15.014797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:15.120209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:15.120149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:15.198659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:15.198626 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-kt5jf_3e916d2c-1680-46eb-8877-f7b20ef70b81/kube-storage-version-migrator-operator/0.log" Sep 10 13:31:15.221308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:15.221279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:15.322340 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:15.322304 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:16.134637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:16.134601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:16.371977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:16.371939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:16.491214 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:16.491146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:16.892253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:16.892222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:17.120639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:17.120612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:17.359232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:17.359197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:17.514004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:17.513976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:17.712502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:17.712446 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:17.955638 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:17.955611 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:18.097234 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.097212 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:18.225048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.225021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:18.349569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.349489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:18.711333 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.711272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:31:18.711333 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.711322 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:31:18.711744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.711697 2559 scope.go:117] "RemoveContainer" containerID="080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094" Sep 10 13:31:18.711870 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:18.711848 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgjcm_openshift-console-operator(6aeb68d5-6782-45d0-8aff-fa15cd8e9266)\"" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" podUID="6aeb68d5-6782-45d0-8aff-fa15cd8e9266" Sep 10 13:31:18.838431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.838398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:18.838529 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:18.838504 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 13:31:18.838568 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:31:18.838560 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls podName:213d309e-3e5d-4df8-9ea1-a3f9010e85ab nodeName:}" failed. No retries permitted until 2026-09-10 13:31:26.838545989 +0000 UTC m=+62.448330879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9wl8" (UID: "213d309e-3e5d-4df8-9ea1-a3f9010e85ab") : secret "insights-runtime-extractor-tls" not found Sep 10 13:31:18.928848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:18.928816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:19.356151 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:19.356124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:19.508408 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:19.508386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:19.717798 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:19.717733 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:19.818057 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:19.818030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:20.888461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:20.888420 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:20.955281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:20.955248 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:21.278639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:21.278612 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:21.492198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:21.492167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:21.685918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:21.685867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:21.879701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:21.879678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:21.956393 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:21.956326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:22.084260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:22.084234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:23.288753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:23.288720 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:23.494413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:23.494386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:24.067261 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:24.067237 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qrztv" Sep 10 13:31:25.167448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:25.167398 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:25.299356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:25.299316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:25.396551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:25.396511 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:26.900652 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:26.900611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:26.903240 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:26.903204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/213d309e-3e5d-4df8-9ea1-a3f9010e85ab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9wl8\" (UID: \"213d309e-3e5d-4df8-9ea1-a3f9010e85ab\") " pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:27.005904 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:27.005877 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2d75h\"" Sep 10 13:31:27.014090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:27.014070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9wl8" Sep 10 13:31:27.294530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:27.294499 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9wl8"] Sep 10 13:31:27.298208 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:27.298186 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213d309e_3e5d_4df8_9ea1_a3f9010e85ab.slice/crio-b258d709df0a47697a63d1f66148fbe5f2374ba3f2a6b0280e1069bf03c59c8e WatchSource:0}: Error finding container b258d709df0a47697a63d1f66148fbe5f2374ba3f2a6b0280e1069bf03c59c8e: Status 404 returned error can't find the container with id b258d709df0a47697a63d1f66148fbe5f2374ba3f2a6b0280e1069bf03c59c8e Sep 10 13:31:28.220453 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:28.220425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9wl8" event={"ID":"213d309e-3e5d-4df8-9ea1-a3f9010e85ab","Type":"ContainerStarted","Data":"95954b04231eb8a16474f1ae653204b57f3a869cbfdf7e4989040cc94522c15d"} Sep 10 13:31:28.220887 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:28.220482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9wl8" event={"ID":"213d309e-3e5d-4df8-9ea1-a3f9010e85ab","Type":"ContainerStarted","Data":"b258d709df0a47697a63d1f66148fbe5f2374ba3f2a6b0280e1069bf03c59c8e"} Sep 10 13:31:29.225878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:29.225846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9wl8" event={"ID":"213d309e-3e5d-4df8-9ea1-a3f9010e85ab","Type":"ContainerStarted","Data":"fee6426e3b67f813d4fee144168095a6e77248dd9ed30dbbd822e8abe18cb28f"} Sep 10 13:31:30.124076 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.124038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:30.126431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.126406 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"image-registry-5d66d65f69-rsshw\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:30.225038 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.225008 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:30.225183 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.225164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:30.225257 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.225196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:30.227753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.227723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dvsk4\" (UID: \"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:30.228144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.227832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9936248a-cfed-420b-914f-b5ebc63997ea-metrics-tls\") pod \"dns-default-ccxnk\" (UID: \"9936248a-cfed-420b-914f-b5ebc63997ea\") " pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:30.228144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.227927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b28758f1-b930-4ef4-8371-b2cb8c6e4ef4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-49wvj\" (UID: \"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:30.325954 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.325922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:30.326128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.326001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:30.326128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.326077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:30.326806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.326779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dbe2dc5-3878-4994-be23-bcedf15db377-service-ca-bundle\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:30.328468 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.328444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dbe2dc5-3878-4994-be23-bcedf15db377-metrics-certs\") pod \"router-default-676fd4f68-q9ztc\" (UID: \"0dbe2dc5-3878-4994-be23-bcedf15db377\") " pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:30.328587 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.328564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7821b6a2-3699-4673-a5ef-b69541733215-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-97mwn\" (UID: \"7821b6a2-3699-4673-a5ef-b69541733215\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:30.398616 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.398558 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tn4wt\"" Sep 10 13:31:30.401488 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.401471 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9764\"" Sep 10 13:31:30.402528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.402510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:30.410399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.410378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:30.434662 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.434640 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dwcmz\"" Sep 10 13:31:30.442262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.442233 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" Sep 10 13:31:30.457575 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.457468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jkzd4\"" Sep 10 13:31:30.465013 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.464858 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" Sep 10 13:31:30.535886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.535859 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xzcxz\"" Sep 10 13:31:30.543015 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.542995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:30.546868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.546850 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5dzcw\"" Sep 10 13:31:30.555008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.554981 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" Sep 10 13:31:30.627747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.627723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:31:30.632460 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.632440 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 13:31:30.640798 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.640779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7954047b-1d2d-4a68-8df2-db357abdf494-original-pull-secret\") pod \"global-pull-secret-syncer-kh6q9\" (UID: \"7954047b-1d2d-4a68-8df2-db357abdf494\") " pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:31:30.682023 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.681964 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:31:30.684159 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.684141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ccxnk"] Sep 10 13:31:30.685579 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:30.685557 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a86429b_d86d_4871_984b_f4ee5212f950.slice/crio-cd1dedaac7a810b33aab9aac220c8935e640ce69bf729304a7f5d7204dab2a5c WatchSource:0}: Error finding container cd1dedaac7a810b33aab9aac220c8935e640ce69bf729304a7f5d7204dab2a5c: Status 404 returned error can't find the container with id cd1dedaac7a810b33aab9aac220c8935e640ce69bf729304a7f5d7204dab2a5c Sep 10 13:31:30.687021 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:30.686988 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9936248a_cfed_420b_914f_b5ebc63997ea.slice/crio-4a42930e97223e07223e7faea2da675a28e1b16168e4903505aa8669b8bfeed2 WatchSource:0}: Error finding container 4a42930e97223e07223e7faea2da675a28e1b16168e4903505aa8669b8bfeed2: Status 404 returned error can't find the container with id 4a42930e97223e07223e7faea2da675a28e1b16168e4903505aa8669b8bfeed2 Sep 10 13:31:30.728430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.728410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:31:30.731016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.731000 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 13:31:30.733029 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.733015 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kh6q9" Sep 10 13:31:30.739573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.739554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4"] Sep 10 13:31:30.741086 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.741061 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a2afc9-2500-44a2-9f17-b886c4ec7142-metrics-certs\") pod \"network-metrics-daemon-c5z5r\" (UID: \"22a2afc9-2500-44a2-9f17-b886c4ec7142\") " pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:31:30.745129 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:30.745108 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16dc7a4_8ad9_4e42_91aa_a9ac7ab00ae9.slice/crio-972c8e6435aae723f954cf32cb9236ae6f1024c8f615ab81d0373aefe89ab238 WatchSource:0}: Error finding container 972c8e6435aae723f954cf32cb9236ae6f1024c8f615ab81d0373aefe89ab238: Status 404 returned error can't find the container with id 972c8e6435aae723f954cf32cb9236ae6f1024c8f615ab81d0373aefe89ab238 Sep 10 13:31:30.762775 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.762716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-49wvj"] Sep 10 13:31:30.767171 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:30.767147 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28758f1_b930_4ef4_8371_b2cb8c6e4ef4.slice/crio-35ca28a1cfc43ea74aea1f92836e46ffcf89847f0447756607e3f90e15640ae7 WatchSource:0}: Error finding container 35ca28a1cfc43ea74aea1f92836e46ffcf89847f0447756607e3f90e15640ae7: Status 404 returned error can't find the container with id 35ca28a1cfc43ea74aea1f92836e46ffcf89847f0447756607e3f90e15640ae7 Sep 10 13:31:30.833670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.833647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-676fd4f68-q9ztc"] Sep 10 13:31:30.835024 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:30.835000 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbe2dc5_3878_4994_be23_bcedf15db377.slice/crio-984ec26f269cbffb6c83c74fab7ac9ee51d0ca0059f142c115af7293d91cd027 WatchSource:0}: Error finding container 984ec26f269cbffb6c83c74fab7ac9ee51d0ca0059f142c115af7293d91cd027: Status 404 returned error can't find the container with id 984ec26f269cbffb6c83c74fab7ac9ee51d0ca0059f142c115af7293d91cd027 Sep 10 13:31:30.840020 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:30.839998 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn"] Sep 10 13:31:31.008071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.008037 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kh6q9"] Sep 10 13:31:31.010776 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:31.010752 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7954047b_1d2d_4a68_8df2_db357abdf494.slice/crio-661efd3c3d23d7c0217a6cc3bdf8a0eb3bcf435dcdcfc84f879eccf832812bb8 WatchSource:0}: Error finding container 661efd3c3d23d7c0217a6cc3bdf8a0eb3bcf435dcdcfc84f879eccf832812bb8: Status 404 returned error can't find the container with id 661efd3c3d23d7c0217a6cc3bdf8a0eb3bcf435dcdcfc84f879eccf832812bb8 Sep 10 13:31:31.027605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.027580 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nc4jc\"" Sep 10 13:31:31.035620 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.035599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c5z5r" Sep 10 13:31:31.236437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.236399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9wl8" event={"ID":"213d309e-3e5d-4df8-9ea1-a3f9010e85ab","Type":"ContainerStarted","Data":"41104a76dab1c9ff89ddd226048d1b8f5f1678012e26c6ba1b3daa1755eb490c"} Sep 10 13:31:31.238281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.238254 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kh6q9" event={"ID":"7954047b-1d2d-4a68-8df2-db357abdf494","Type":"ContainerStarted","Data":"661efd3c3d23d7c0217a6cc3bdf8a0eb3bcf435dcdcfc84f879eccf832812bb8"} Sep 10 13:31:31.239594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.239566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" event={"ID":"7821b6a2-3699-4673-a5ef-b69541733215","Type":"ContainerStarted","Data":"32aa60d649ed971537f5b1a2006d42e4a72a48f79e502fc0b9b3d5c3e0b65dc0"} Sep 10 13:31:31.241293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.241264 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-676fd4f68-q9ztc" event={"ID":"0dbe2dc5-3878-4994-be23-bcedf15db377","Type":"ContainerStarted","Data":"eda28f70d1805b9495357496aa9a43d85b185a2e8a1098f05c90364de7594e8e"} Sep 10 13:31:31.241430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.241306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-676fd4f68-q9ztc" event={"ID":"0dbe2dc5-3878-4994-be23-bcedf15db377","Type":"ContainerStarted","Data":"984ec26f269cbffb6c83c74fab7ac9ee51d0ca0059f142c115af7293d91cd027"} Sep 10 13:31:31.242800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.242768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" event={"ID":"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9","Type":"ContainerStarted","Data":"972c8e6435aae723f954cf32cb9236ae6f1024c8f615ab81d0373aefe89ab238"} Sep 10 13:31:31.244132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.244105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccxnk" event={"ID":"9936248a-cfed-420b-914f-b5ebc63997ea","Type":"ContainerStarted","Data":"4a42930e97223e07223e7faea2da675a28e1b16168e4903505aa8669b8bfeed2"} Sep 10 13:31:31.245402 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.245359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" event={"ID":"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4","Type":"ContainerStarted","Data":"35ca28a1cfc43ea74aea1f92836e46ffcf89847f0447756607e3f90e15640ae7"} Sep 10 13:31:31.246959 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.246937 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" event={"ID":"4a86429b-d86d-4871-984b-f4ee5212f950","Type":"ContainerStarted","Data":"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255"} Sep 10 13:31:31.247066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.246978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" event={"ID":"4a86429b-d86d-4871-984b-f4ee5212f950","Type":"ContainerStarted","Data":"cd1dedaac7a810b33aab9aac220c8935e640ce69bf729304a7f5d7204dab2a5c"} Sep 10 13:31:31.247127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.247090 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:31.293868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.293803 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v9wl8" podStartSLOduration=17.239610121 podStartE2EDuration="20.293789715s" totalImagesPullingTime="3.054179594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:31.26370992 +0000 UTC m=+66.873494834" watchObservedRunningTime="2026-09-10 13:31:31.293789715 +0000 UTC m=+66.903574628" Sep 10 13:31:31.294032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.294009 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" podStartSLOduration=66.294002152 podStartE2EDuration="1m6.294002152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:31.292612904 +0000 UTC m=+66.902397850" watchObservedRunningTime="2026-09-10 13:31:31.294002152 +0000 UTC m=+66.903787066" Sep 10 13:31:31.319281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.319259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c5z5r"] Sep 10 13:31:31.322958 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:31.322937 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a2afc9_2500_44a2_9f17_b886c4ec7142.slice/crio-a67b6e1875cccadb216f98fc2f61b0b9f6885c78e12d0a7b4c8ef22b0a8f9ffb WatchSource:0}: Error finding container a67b6e1875cccadb216f98fc2f61b0b9f6885c78e12d0a7b4c8ef22b0a8f9ffb: Status 404 returned error can't find the container with id a67b6e1875cccadb216f98fc2f61b0b9f6885c78e12d0a7b4c8ef22b0a8f9ffb Sep 10 13:31:31.326398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.326330 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-676fd4f68-q9ztc" podStartSLOduration=46.326313424 podStartE2EDuration="46.326313424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:31.326071435 +0000 UTC m=+66.935856349" watchObservedRunningTime="2026-09-10 13:31:31.326313424 +0000 UTC m=+66.936098342" Sep 10 13:31:31.543834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.543801 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:31.546570 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:31.546507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:32.018898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:32.018866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:32.252435 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:32.252400 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5z5r" event={"ID":"22a2afc9-2500-44a2-9f17-b886c4ec7142","Type":"ContainerStarted","Data":"a67b6e1875cccadb216f98fc2f61b0b9f6885c78e12d0a7b4c8ef22b0a8f9ffb"} Sep 10 13:31:32.253303 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:32.253071 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:32.254175 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:32.254152 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-676fd4f68-q9ztc" Sep 10 13:31:33.912755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:33.912729 2559 scope.go:117] "RemoveContainer" containerID="080458c48480713fa3daeac999283e2478373e0904821141d8a055027e363094" Sep 10 13:31:34.265812 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.265777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" event={"ID":"7821b6a2-3699-4673-a5ef-b69541733215","Type":"ContainerStarted","Data":"8ef2eddba2688cc1c9b32cd3f86743f6660534d91dd2b79cd248cde5c75cc086"} Sep 10 13:31:34.265971 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.265844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" event={"ID":"7821b6a2-3699-4673-a5ef-b69541733215","Type":"ContainerStarted","Data":"66db6d5d491206fc45004a432aac505f021581be05441bc884d72402480ef86d"} Sep 10 13:31:34.268212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.268106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5z5r" event={"ID":"22a2afc9-2500-44a2-9f17-b886c4ec7142","Type":"ContainerStarted","Data":"6d74dc6df2e9f9666c2b459106c98274417bfe8879f2ce6aced38fbb6cb007b5"} Sep 10 13:31:34.268212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.268159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c5z5r" event={"ID":"22a2afc9-2500-44a2-9f17-b886c4ec7142","Type":"ContainerStarted","Data":"41bec7ee4c0f9e02b81c0f736ad9dc8553b97a43789257ebaefd255c3dd7b761"} Sep 10 13:31:34.270737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.270714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccxnk" event={"ID":"9936248a-cfed-420b-914f-b5ebc63997ea","Type":"ContainerStarted","Data":"e66e2d8206d1f645c10f200423b2ea6451d4de669f2f2fd9ba6407e0a5fd9c50"} Sep 10 13:31:34.270847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.270758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ccxnk" event={"ID":"9936248a-cfed-420b-914f-b5ebc63997ea","Type":"ContainerStarted","Data":"c9962282c7c69476766c63f68172db29dc7480d815a7c0b7b0cba6c8fcfcee55"} Sep 10 13:31:34.270847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.270822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:34.272543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.272515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" event={"ID":"b28758f1-b930-4ef4-8371-b2cb8c6e4ef4","Type":"ContainerStarted","Data":"72437946dbaabb3913d2a11c01c1a1468341a2f1722e1d1239348f600ce15bf1"} Sep 10 13:31:34.274722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.274702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:31:34.274864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.274822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" event={"ID":"6aeb68d5-6782-45d0-8aff-fa15cd8e9266","Type":"ContainerStarted","Data":"8da19cb334b09413702945d0d449fcf1bd850ce2ebd2ce441f5834415b0d4a3c"} Sep 10 13:31:34.275165 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.275140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:31:34.296934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.296636 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-97mwn" podStartSLOduration=46.455246778 podStartE2EDuration="49.296618654s" totalImagesPullingTime="2.841371876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:34.295278906 +0000 UTC m=+69.905063821" watchObservedRunningTime="2026-09-10 13:31:34.296618654 +0000 UTC m=+69.906403568" Sep 10 13:31:34.366415 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.366340 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-49wvj" podStartSLOduration=46.346683323 podStartE2EDuration="49.366320229s" totalImagesPullingTime="3.019636906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:34.365005887 +0000 UTC m=+69.974790805" watchObservedRunningTime="2026-09-10 13:31:34.366320229 +0000 UTC m=+69.976105143" Sep 10 13:31:34.370253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.367659 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ccxnk" podStartSLOduration=33.33693439 podStartE2EDuration="36.367648982s" totalImagesPullingTime="3.030714592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:34.333658861 +0000 UTC m=+69.943443776" watchObservedRunningTime="2026-09-10 13:31:34.367648982 +0000 UTC m=+69.977433896" Sep 10 13:31:34.392974 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.392719 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" podStartSLOduration=39.465652128 podStartE2EDuration="49.392702499s" totalImagesPullingTime="9.927050371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:34.39207668 +0000 UTC m=+70.001861620" watchObservedRunningTime="2026-09-10 13:31:34.392702499 +0000 UTC m=+70.002487414" Sep 10 13:31:34.427693 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.427653 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c5z5r" podStartSLOduration=67.032939781 podStartE2EDuration="1m9.427638638s" totalImagesPullingTime="2.394698857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:34.425530887 +0000 UTC m=+70.035315801" watchObservedRunningTime="2026-09-10 13:31:34.427638638 +0000 UTC m=+70.037423551" Sep 10 13:31:34.545430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.545330 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:34.683191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.683160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-zgjcm" Sep 10 13:31:34.685670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.685638 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:34.888839 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.888757 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-wtptv"] Sep 10 13:31:34.907084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.907060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:34.910544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.910521 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 13:31:34.910544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.910577 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 13:31:34.910544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.910614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wtptv"] Sep 10 13:31:34.910544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.910736 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kpnqd\"" Sep 10 13:31:34.965183 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:34.965157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnx4k\" (UniqueName: \"kubernetes.io/projected/669c481b-fdf8-45f2-b199-a34dfd1dcc52-kube-api-access-pnx4k\") pod \"downloads-85d98c998f-wtptv\" (UID: \"669c481b-fdf8-45f2-b199-a34dfd1dcc52\") " pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:35.066472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:35.066441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnx4k\" (UniqueName: \"kubernetes.io/projected/669c481b-fdf8-45f2-b199-a34dfd1dcc52-kube-api-access-pnx4k\") pod \"downloads-85d98c998f-wtptv\" (UID: \"669c481b-fdf8-45f2-b199-a34dfd1dcc52\") " pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:35.080916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:35.080890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnx4k\" (UniqueName: \"kubernetes.io/projected/669c481b-fdf8-45f2-b199-a34dfd1dcc52-kube-api-access-pnx4k\") pod \"downloads-85d98c998f-wtptv\" (UID: \"669c481b-fdf8-45f2-b199-a34dfd1dcc52\") " pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:35.221513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:35.221436 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:36.239232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:36.239156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:36.336735 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:36.336683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:36.583878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:36.583841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:36.783071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:36.783038 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:37.008286 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.008261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:37.290986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.290868 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kh6q9" event={"ID":"7954047b-1d2d-4a68-8df2-db357abdf494","Type":"ContainerStarted","Data":"dc7ca2a0df4305e61ebfee1c1e9e831bb38824f1d3b3f69f9e852f5a819860fb"} Sep 10 13:31:37.292762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.292727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" event={"ID":"d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9","Type":"ContainerStarted","Data":"55ed0239ac70897e4cea66f8cb71f2f6fc31927b4afab32e820c5f8d705b19ab"} Sep 10 13:31:37.298483 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.298461 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wtptv"] Sep 10 13:31:37.302047 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:37.302015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669c481b_fdf8_45f2_b199_a34dfd1dcc52.slice/crio-e98b76402a3802482366fb4a799d63e9a29ae52de337614f0db726c16c4f8977 WatchSource:0}: Error finding container e98b76402a3802482366fb4a799d63e9a29ae52de337614f0db726c16c4f8977: Status 404 returned error can't find the container with id e98b76402a3802482366fb4a799d63e9a29ae52de337614f0db726c16c4f8977 Sep 10 13:31:37.332899 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.332852 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kh6q9" podStartSLOduration=66.313812554 podStartE2EDuration="1m12.332838645s" totalImagesPullingTime="6.019026091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:37.311721294 +0000 UTC m=+72.921506207" watchObservedRunningTime="2026-09-10 13:31:37.332838645 +0000 UTC m=+72.942623620" Sep 10 13:31:37.333030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.332965 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dvsk4" podStartSLOduration=46.044796274 podStartE2EDuration="52.332957786s" totalImagesPullingTime="6.288161512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:37.332558762 +0000 UTC m=+72.942343676" watchObservedRunningTime="2026-09-10 13:31:37.332957786 +0000 UTC m=+72.942742699" Sep 10 13:31:37.496661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.496626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:37.677927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.677834 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:37.685286 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.685244 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68"] Sep 10 13:31:37.711934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.711893 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68"] Sep 10 13:31:37.712112 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.712007 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:37.714848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.714806 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q4mkr\"" Sep 10 13:31:37.714981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.714867 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 13:31:37.790324 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.790293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c521425-b190-49ee-bc34-3c885fce1786-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fst68\" (UID: \"8c521425-b190-49ee-bc34-3c885fce1786\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:37.838023 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.837983 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:37.891388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.891338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c521425-b190-49ee-bc34-3c885fce1786-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fst68\" (UID: \"8c521425-b190-49ee-bc34-3c885fce1786\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:37.893705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.893687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8c521425-b190-49ee-bc34-3c885fce1786-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-fst68\" (UID: \"8c521425-b190-49ee-bc34-3c885fce1786\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:37.946075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:37.945987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:38.022107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:38.022069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:38.298634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:38.298598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wtptv" event={"ID":"669c481b-fdf8-45f2-b199-a34dfd1dcc52","Type":"ContainerStarted","Data":"e98b76402a3802482366fb4a799d63e9a29ae52de337614f0db726c16c4f8977"} Sep 10 13:31:38.318012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:38.317962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68"] Sep 10 13:31:38.321576 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:38.321544 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c521425_b190_49ee_bc34_3c885fce1786.slice/crio-d8380c067c76d4d81be7271a61439106d8ebcdad56e53dbff91da5687e61fb47 WatchSource:0}: Error finding container d8380c067c76d4d81be7271a61439106d8ebcdad56e53dbff91da5687e61fb47: Status 404 returned error can't find the container with id d8380c067c76d4d81be7271a61439106d8ebcdad56e53dbff91da5687e61fb47 Sep 10 13:31:39.308304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:39.308212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" event={"ID":"8c521425-b190-49ee-bc34-3c885fce1786","Type":"ContainerStarted","Data":"d8380c067c76d4d81be7271a61439106d8ebcdad56e53dbff91da5687e61fb47"} Sep 10 13:31:39.731157 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:39.731074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:39.814190 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:39.814154 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:40.314255 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:40.314218 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" event={"ID":"8c521425-b190-49ee-bc34-3c885fce1786","Type":"ContainerStarted","Data":"7f415d60eb39729f9c7860cce744d7ffd3d1e2d177fd7f2ba9ccc2dc354fa91d"} Sep 10 13:31:40.314737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:40.314510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:40.319897 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:40.319875 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" Sep 10 13:31:40.361258 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:40.361126 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-fst68" podStartSLOduration=1.704265887 podStartE2EDuration="3.361108739s" totalImagesPullingTime="1.656842852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:40.360975878 +0000 UTC m=+75.970760791" watchObservedRunningTime="2026-09-10 13:31:40.361108739 +0000 UTC m=+75.970893655" Sep 10 13:31:40.725410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:40.725313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:41.153773 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:41.153741 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9htdb" Sep 10 13:31:42.234669 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:42.234629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:42.341185 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:42.341137 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:44.283518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:44.283491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ccxnk" Sep 10 13:31:46.072477 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:46.072423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:46.520892 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:46.520807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:46.707287 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:46.707240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:48.034150 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.034001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:48.347439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.347269 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:48.553715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.553678 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4q6cj"] Sep 10 13:31:48.558551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.558528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.569223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.569199 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 13:31:48.569842 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.569816 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 13:31:48.573508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.573482 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 13:31:48.574576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.574553 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 13:31:48.576780 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.576759 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tt9jm\"" Sep 10 13:31:48.687982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.687902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-textfile\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.687982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.687940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-sys\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.687982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.687980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8g6x\" (UniqueName: \"kubernetes.io/projected/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-kube-api-access-q8g6x\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-metrics-client-ca\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-wtmp\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-root\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.688479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.688305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-tls\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788714 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788668 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-metrics-client-ca\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-wtmp\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-root\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-tls\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-textfile\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.788913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-sys\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.788973 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-sys\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.789027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-root\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.789062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-wtmp\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.789117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8g6x\" (UniqueName: \"kubernetes.io/projected/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-kube-api-access-q8g6x\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.789576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.789793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.789775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-textfile\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.790217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.790178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-metrics-client-ca\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.791665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.791623 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.791775 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.791754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-node-exporter-tls\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.804620 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.804598 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8g6x\" (UniqueName: \"kubernetes.io/projected/41ab9a90-3f87-4d36-bf7f-0de5c169e3f3-kube-api-access-q8g6x\") pod \"node-exporter-4q6cj\" (UID: \"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3\") " pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.870548 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:48.870514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4q6cj" Sep 10 13:31:48.883316 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:48.883279 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ab9a90_3f87_4d36_bf7f_0de5c169e3f3.slice/crio-5bd549df4892d9056c91cbc8622729c619355d8b9d0ab877749fcdd1ecb249b7 WatchSource:0}: Error finding container 5bd549df4892d9056c91cbc8622729c619355d8b9d0ab877749fcdd1ecb249b7: Status 404 returned error can't find the container with id 5bd549df4892d9056c91cbc8622729c619355d8b9d0ab877749fcdd1ecb249b7 Sep 10 13:31:49.355103 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:49.355016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4q6cj" event={"ID":"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3","Type":"ContainerStarted","Data":"5bd549df4892d9056c91cbc8622729c619355d8b9d0ab877749fcdd1ecb249b7"} Sep 10 13:31:49.628948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:49.628855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:49.766605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:49.766553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:50.407995 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:50.407955 2559 patch_prober.go:28] interesting pod/image-registry-5d66d65f69-rsshw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 13:31:50.408407 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:50.408025 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:31:50.489649 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:50.489607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:50.635747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:50.635696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:51.563338 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:51.563290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:51.836154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:51.836073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:52.258597 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:52.258566 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:31:52.521717 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:52.521577 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:52.665309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:52.665254 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:53.169554 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:53.169516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:53.334927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:53.334870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:53.579267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:53.579227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:53.679773 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:53.679737 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:54.186679 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:54.186643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:54.919222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:54.919192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:55.850097 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:55.850053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:55.940263 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:55.940225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:56.391726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:56.391638 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wtptv" event={"ID":"669c481b-fdf8-45f2-b199-a34dfd1dcc52","Type":"ContainerStarted","Data":"ac3fb697b194b584e966f8c259ba72026d752b590b9df59fca55e584ac7f4b84"} Sep 10 13:31:56.392175 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:56.392155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:56.399058 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:56.399032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-wtptv" Sep 10 13:31:56.422133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:56.422015 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-wtptv" podStartSLOduration=3.664973761 podStartE2EDuration="22.42199502s" totalImagesPullingTime="18.757021259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:56.42180821 +0000 UTC m=+92.031593123" watchObservedRunningTime="2026-09-10 13:31:56.42199502 +0000 UTC m=+92.031779934" Sep 10 13:31:56.875990 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:56.875957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:57.030430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.030391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:57.132091 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.132053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:57.339646 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.339567 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:31:57.342261 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.342226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:57.360092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.360071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.365209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.365189 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 13:31:57.365679 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.365474 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 13:31:57.365679 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.365562 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5dh5t\"" Sep 10 13:31:57.366543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.366523 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 13:31:57.367541 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkdf\" (UniqueName: \"kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367639 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367689 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367772 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.367896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.367876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.393140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.393106 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 13:31:57.394642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.394451 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 13:31:57.395710 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.395629 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:31:57.402642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.402598 2559 generic.go:356] "Generic (PLEG): container finished" podID="41ab9a90-3f87-4d36-bf7f-0de5c169e3f3" containerID="4f701f2473e85d37bad476332e077c8900d22d534636786b10964ee7abc8deaa" exitCode=0 Sep 10 13:31:57.402748 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.402718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4q6cj" event={"ID":"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3","Type":"ContainerDied","Data":"4f701f2473e85d37bad476332e077c8900d22d534636786b10964ee7abc8deaa"} Sep 10 13:31:57.421486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.421459 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 13:31:57.468638 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.468761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.468761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkdf\" (UniqueName: \"kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.468935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.468935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.468935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.469084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.468953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.469859 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.469340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.469859 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.469817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.470400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.470330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.471241 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.471218 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.472680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.472650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.473043 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.473018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.502445 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.502417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkdf\" (UniqueName: \"kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf\") pod \"console-7cf94f688b-9x8q7\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.525215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.525179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:57.671741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.671694 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:31:57.972866 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:57.972787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:31:57.976632 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:31:57.976605 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8527d2c_ecb8_498f_86b2_d5e2d2ea71d3.slice/crio-56282e67d7305235cb1041f3ae689772dd59e06a099e485e7712cb4a8ed4de04 WatchSource:0}: Error finding container 56282e67d7305235cb1041f3ae689772dd59e06a099e485e7712cb4a8ed4de04: Status 404 returned error can't find the container with id 56282e67d7305235cb1041f3ae689772dd59e06a099e485e7712cb4a8ed4de04 Sep 10 13:31:58.017711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.017677 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:31:58.410063 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.410016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4q6cj" event={"ID":"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3","Type":"ContainerStarted","Data":"0080469aeadafd14862c01806a5917ef17431bb1e1207776b9d36bd85d0c2401"} Sep 10 13:31:58.410655 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.410088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4q6cj" event={"ID":"41ab9a90-3f87-4d36-bf7f-0de5c169e3f3","Type":"ContainerStarted","Data":"1804719d25182e38794280afb0777d719a53576d155166109968a00b2a25c6ff"} Sep 10 13:31:58.411837 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.411801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf94f688b-9x8q7" event={"ID":"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3","Type":"ContainerStarted","Data":"56282e67d7305235cb1041f3ae689772dd59e06a099e485e7712cb4a8ed4de04"} Sep 10 13:31:58.456106 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.456046 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4q6cj" podStartSLOduration=2.542438699 podStartE2EDuration="10.456025324s" totalImagesPullingTime="7.835781929s" totalInitContainerRuntime="77.804696ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:58.453606429 +0000 UTC m=+94.063391343" watchObservedRunningTime="2026-09-10 13:31:58.456025324 +0000 UTC m=+94.065810238" Sep 10 13:31:58.715847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.715755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:31:58.807293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:31:58.807261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.390841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:02.390802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:02.535003 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:02.534942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:03.297778 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:03.297742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:03.439843 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:03.439757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf94f688b-9x8q7" event={"ID":"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3","Type":"ContainerStarted","Data":"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6"} Sep 10 13:32:03.455831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:03.455798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:03.473172 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:03.473120 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf94f688b-9x8q7" podStartSLOduration=1.315914134 podStartE2EDuration="6.473103045s" totalImagesPullingTime="5.157188911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:03.472745001 +0000 UTC m=+99.082529913" watchObservedRunningTime="2026-09-10 13:32:03.473103045 +0000 UTC m=+99.082887959" Sep 10 13:32:05.003326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:05.003287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:05.103346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:05.103309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:05.213184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:05.213150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.671877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:07.671835 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:32:07.672352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:07.671944 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:32:07.672352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:07.672187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:07.678416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:07.677907 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:32:08.465482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:08.465451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:32:09.136399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:09.136341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:09.244231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:09.244203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:09.764445 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:09.764409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:09.928554 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:09.928518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:12.197428 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:12.197386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:23.042800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.042763 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" containerName="registry" containerID="cri-o://f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255" gracePeriod=30 Sep 10 13:32:23.330450 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.330426 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:32:23.357781 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.357759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.357903 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.357813 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.357903 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.357842 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.357903 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.357867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.358227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.358275 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.358312 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.358346 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7swtn\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn\") pod \"4a86429b-d86d-4871-984b-f4ee5212f950\" (UID: \"4a86429b-d86d-4871-984b-f4ee5212f950\") " Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.358347 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.359054 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-registry-certificates\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.359192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.359161 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:32:23.360988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.360916 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:32:23.361294 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.361267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:32:23.361495 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.361461 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:32:23.361943 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.361571 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn" (OuterVolumeSpecName: "kube-api-access-7swtn") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "kube-api-access-7swtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:32:23.362326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.362288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:32:23.369347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.369318 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a86429b-d86d-4871-984b-f4ee5212f950" (UID: "4a86429b-d86d-4871-984b-f4ee5212f950"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:32:23.459485 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459459 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-installation-pull-secrets\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459485 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459482 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a86429b-d86d-4871-984b-f4ee5212f950-image-registry-private-configuration\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459493 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-registry-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459502 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-bound-sa-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459511 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a86429b-d86d-4871-984b-f4ee5212f950-trusted-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459519 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swtn\" (UniqueName: \"kubernetes.io/projected/4a86429b-d86d-4871-984b-f4ee5212f950-kube-api-access-7swtn\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.459595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.459527 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a86429b-d86d-4871-984b-f4ee5212f950-ca-trust-extracted\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:32:23.519855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.519832 2559 generic.go:356] "Generic (PLEG): container finished" podID="4a86429b-d86d-4871-984b-f4ee5212f950" containerID="f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255" exitCode=0 Sep 10 13:32:23.519932 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.519874 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" Sep 10 13:32:23.519972 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.519923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" event={"ID":"4a86429b-d86d-4871-984b-f4ee5212f950","Type":"ContainerDied","Data":"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255"} Sep 10 13:32:23.520008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.519969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66d65f69-rsshw" event={"ID":"4a86429b-d86d-4871-984b-f4ee5212f950","Type":"ContainerDied","Data":"cd1dedaac7a810b33aab9aac220c8935e640ce69bf729304a7f5d7204dab2a5c"} Sep 10 13:32:23.520008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.519993 2559 scope.go:117] "RemoveContainer" containerID="f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255" Sep 10 13:32:23.531253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.531233 2559 scope.go:117] "RemoveContainer" containerID="f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255" Sep 10 13:32:23.531589 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:32:23.531557 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255\": container with ID starting with f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255 not found: ID does not exist" containerID="f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255" Sep 10 13:32:23.531674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.531599 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255"} err="failed to get container status \"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255\": rpc error: code = NotFound desc = could not find container \"f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255\": container with ID starting with f949dff4f443f48734dc95984d0795b4e5b1c5bf8821e881ca440f17ef258255 not found: ID does not exist" Sep 10 13:32:23.547947 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.547897 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:32:23.555559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:23.555540 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d66d65f69-rsshw"] Sep 10 13:32:24.750480 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:24.750445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.889581 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:24.889557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:24.916337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:24.916309 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" path="/var/lib/kubelet/pods/4a86429b-d86d-4871-984b-f4ee5212f950/volumes" Sep 10 13:32:31.554760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:31.554725 2559 generic.go:356] "Generic (PLEG): container finished" podID="2fd2d410-6c72-4e23-8d17-d1554cbe69e8" containerID="e226b4d21bc17623e79d399b6827c6cca589960a154dbce9ef10b360d91c3871" exitCode=0 Sep 10 13:32:31.555210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:31.554805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" event={"ID":"2fd2d410-6c72-4e23-8d17-d1554cbe69e8","Type":"ContainerDied","Data":"e226b4d21bc17623e79d399b6827c6cca589960a154dbce9ef10b360d91c3871"} Sep 10 13:32:31.555210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:31.555158 2559 scope.go:117] "RemoveContainer" containerID="e226b4d21bc17623e79d399b6827c6cca589960a154dbce9ef10b360d91c3871" Sep 10 13:32:32.560839 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:32.560807 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-dkbfp" event={"ID":"2fd2d410-6c72-4e23-8d17-d1554cbe69e8","Type":"ContainerStarted","Data":"f1e42f1133c3bac07533a06989ff69f05017563764891120b0cd37038ada5523"} Sep 10 13:32:35.574148 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.574110 2559 generic.go:356] "Generic (PLEG): container finished" podID="3e916d2c-1680-46eb-8877-f7b20ef70b81" containerID="54593538e75af071a1970486af26097f04d90343ef96a7c4f886c71cb21f0b02" exitCode=0 Sep 10 13:32:35.574588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.574184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" event={"ID":"3e916d2c-1680-46eb-8877-f7b20ef70b81","Type":"ContainerDied","Data":"54593538e75af071a1970486af26097f04d90343ef96a7c4f886c71cb21f0b02"} Sep 10 13:32:35.574588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.574522 2559 scope.go:117] "RemoveContainer" containerID="54593538e75af071a1970486af26097f04d90343ef96a7c4f886c71cb21f0b02" Sep 10 13:32:35.576162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.576134 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea039094-07d1-4d6b-98b1-04f6134ed357" containerID="b1ac8bd903c23bf4e34f104235771b9df50d4bb73e2170f6e3314c241121ae0c" exitCode=0 Sep 10 13:32:35.576268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.576172 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" event={"ID":"ea039094-07d1-4d6b-98b1-04f6134ed357","Type":"ContainerDied","Data":"b1ac8bd903c23bf4e34f104235771b9df50d4bb73e2170f6e3314c241121ae0c"} Sep 10 13:32:35.576474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:35.576459 2559 scope.go:117] "RemoveContainer" containerID="b1ac8bd903c23bf4e34f104235771b9df50d4bb73e2170f6e3314c241121ae0c" Sep 10 13:32:36.581320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:36.581284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8spv" event={"ID":"ea039094-07d1-4d6b-98b1-04f6134ed357","Type":"ContainerStarted","Data":"52c4cd429ca7c5a87ea6b3967b2cbba0718046917b8a882da40afae764786d02"} Sep 10 13:32:36.583216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:36.583189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-kt5jf" event={"ID":"3e916d2c-1680-46eb-8877-f7b20ef70b81","Type":"ContainerStarted","Data":"3a6619d444ee30e1d1d8026f6c4d6ec528fe0c69c472e30d81c37ff3ac1ce62e"} Sep 10 13:32:37.542907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:37.542873 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:37.955805 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:37.955728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.200969 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.200936 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:32:46.201329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.201242 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" containerName="registry" Sep 10 13:32:46.201329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.201254 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" containerName="registry" Sep 10 13:32:46.201329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.201316 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a86429b-d86d-4871-984b-f4ee5212f950" containerName="registry" Sep 10 13:32:46.205676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.205654 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.224572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.224547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:32:46.319518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.319483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.320306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320288 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2fz\" (UniqueName: \"kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320457 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320492 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.320523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.320519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421394 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2fz\" (UniqueName: \"kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.421585 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.421562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.422029 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.422003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.422231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.422196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.422313 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.422256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.422313 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.422256 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.423727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.423707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.423916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.423895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.441447 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.441427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2fz\" (UniqueName: \"kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz\") pod \"console-9477b7b89-lbxc8\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.453956 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.453906 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:46.514887 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.514863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:46.795870 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:46.795837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:32:46.798629 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:32:46.798605 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb524afd6_00fc_4a78_b065_bd9bf027d732.slice/crio-b27892f4b5577e9305ce25a21fba15d8a8b95127c383df9ccf7a930e59d5327f WatchSource:0}: Error finding container b27892f4b5577e9305ce25a21fba15d8a8b95127c383df9ccf7a930e59d5327f: Status 404 returned error can't find the container with id b27892f4b5577e9305ce25a21fba15d8a8b95127c383df9ccf7a930e59d5327f Sep 10 13:32:47.632178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:47.632134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b7b89-lbxc8" event={"ID":"b524afd6-00fc-4a78-b065-bd9bf027d732","Type":"ContainerStarted","Data":"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4"} Sep 10 13:32:47.632593 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:47.632202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b7b89-lbxc8" event={"ID":"b524afd6-00fc-4a78-b065-bd9bf027d732","Type":"ContainerStarted","Data":"b27892f4b5577e9305ce25a21fba15d8a8b95127c383df9ccf7a930e59d5327f"} Sep 10 13:32:47.661473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:47.661421 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9477b7b89-lbxc8" podStartSLOduration=1.661405762 podStartE2EDuration="1.661405762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:47.660560855 +0000 UTC m=+143.270345773" watchObservedRunningTime="2026-09-10 13:32:47.661405762 +0000 UTC m=+143.271190675" Sep 10 13:32:56.515242 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.515202 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:56.515800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.515385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:56.520509 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.520482 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:56.678719 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.678692 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:32:56.744030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.743992 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:32:56.902997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:56.902912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:32:57.066895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:32:57.066864 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:21.767845 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:21.767745 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cf94f688b-9x8q7" podUID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" containerName="console" containerID="cri-o://aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6" gracePeriod=15 Sep 10 13:33:22.004058 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.004035 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:33:22.085739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085665 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085724 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085754 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085775 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxkdf\" (UniqueName: \"kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085829 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.085962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.085875 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert\") pod \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\" (UID: \"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3\") " Sep 10 13:33:22.086249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.086225 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config" (OuterVolumeSpecName: "console-config") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:22.086249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.086233 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:22.086352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.086214 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:22.086352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.086310 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:33:22.087877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.087846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:22.087981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.087917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf" (OuterVolumeSpecName: "kube-api-access-wxkdf") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "kube-api-access-wxkdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:33:22.088025 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.087972 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" (UID: "d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:33:22.186344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186305 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186340 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-oauth-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186350 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkdf\" (UniqueName: \"kubernetes.io/projected/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-kube-api-access-wxkdf\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186360 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-oauth-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186393 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-trusted-ca-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186402 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.186562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.186412 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3-console-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:33:22.776815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.776779 2559 generic.go:356] "Generic (PLEG): container finished" podID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" containerID="aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6" exitCode=0 Sep 10 13:33:22.777244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.776821 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf94f688b-9x8q7" Sep 10 13:33:22.777244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.776831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf94f688b-9x8q7" event={"ID":"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3","Type":"ContainerDied","Data":"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6"} Sep 10 13:33:22.777244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.776870 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf94f688b-9x8q7" event={"ID":"d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3","Type":"ContainerDied","Data":"56282e67d7305235cb1041f3ae689772dd59e06a099e485e7712cb4a8ed4de04"} Sep 10 13:33:22.777244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.776889 2559 scope.go:117] "RemoveContainer" containerID="aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6" Sep 10 13:33:22.785292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.785275 2559 scope.go:117] "RemoveContainer" containerID="aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6" Sep 10 13:33:22.785556 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:33:22.785540 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6\": container with ID starting with aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6 not found: ID does not exist" containerID="aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6" Sep 10 13:33:22.785615 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.785563 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6"} err="failed to get container status \"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6\": rpc error: code = NotFound desc = could not find container \"aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6\": container with ID starting with aadd6fac84616c79c88f6b96e22bb1bc7d239d73fdc4c8c8cc5f44525e8472c6 not found: ID does not exist" Sep 10 13:33:22.799264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.799241 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:33:22.803910 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.803892 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cf94f688b-9x8q7"] Sep 10 13:33:22.917476 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:22.917451 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" path="/var/lib/kubelet/pods/d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3/volumes" Sep 10 13:33:52.833153 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:52.833118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:33:52.980589 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:33:52.980555 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:03.075001 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:03.074967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:03.263996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:03.263964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:04.922162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:04.922124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:05.002502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:05.002468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:09.735185 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:09.735140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:09.835856 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:09.835819 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:21.004424 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.004382 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 13:34:21.004815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.004691 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" containerName="console" Sep 10 13:34:21.004815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.004704 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" containerName="console" Sep 10 13:34:21.004815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.004750 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8527d2c-ecb8-498f-86b2-d5e2d2ea71d3" containerName="console" Sep 10 13:34:21.007695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.007671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.024140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.024115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 13:34:21.079154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.079116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:21.102076 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102230 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnc4\" (UniqueName: \"kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.102448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.102357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.169361 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.169310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:21.203351 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnc4\" (UniqueName: \"kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203625 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203625 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.203721 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.203616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.204145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.204121 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.204326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.204304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.204395 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.204346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.204439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.204349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.205864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.205846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.205966 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.205867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.215587 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.215565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnc4\" (UniqueName: \"kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4\") pod \"console-7658ccd565-vp6f9\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.317074 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.317042 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:21.600161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:21.600097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 13:34:21.602949 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:34:21.602922 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ec93e6_f123_491a_948e_2ab0db48e639.slice/crio-7c02bcd7a4e09a59493eb04d163056cad377af707a15a8e344fa506dab309c21 WatchSource:0}: Error finding container 7c02bcd7a4e09a59493eb04d163056cad377af707a15a8e344fa506dab309c21: Status 404 returned error can't find the container with id 7c02bcd7a4e09a59493eb04d163056cad377af707a15a8e344fa506dab309c21 Sep 10 13:34:22.011956 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:22.011919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7658ccd565-vp6f9" event={"ID":"56ec93e6-f123-491a-948e-2ab0db48e639","Type":"ContainerStarted","Data":"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d"} Sep 10 13:34:22.012309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:22.011973 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7658ccd565-vp6f9" event={"ID":"56ec93e6-f123-491a-948e-2ab0db48e639","Type":"ContainerStarted","Data":"7c02bcd7a4e09a59493eb04d163056cad377af707a15a8e344fa506dab309c21"} Sep 10 13:34:22.036107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:22.036063 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7658ccd565-vp6f9" podStartSLOduration=2.036048321 podStartE2EDuration="2.036048321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:22.034810359 +0000 UTC m=+237.644595272" watchObservedRunningTime="2026-09-10 13:34:22.036048321 +0000 UTC m=+237.645833233" Sep 10 13:34:31.317631 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:31.317589 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:31.318114 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:31.317646 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:31.322502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:31.322478 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:32.055406 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.055354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 13:34:32.116093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.116060 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:34:32.247609 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.247572 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:32.347535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.347464 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:32.476673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.476644 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:32.773632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:32.773607 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:39.944687 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:39.944647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:34:57.139163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.139055 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9477b7b89-lbxc8" podUID="b524afd6-00fc-4a78-b065-bd9bf027d732" containerName="console" containerID="cri-o://a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4" gracePeriod=15 Sep 10 13:34:57.372953 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.372929 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:34:57.547100 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh2fz\" (UniqueName: \"kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547159 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547190 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547250 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547282 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle\") pod \"b524afd6-00fc-4a78-b065-bd9bf027d732\" (UID: \"b524afd6-00fc-4a78-b065-bd9bf027d732\") " Sep 10 13:34:57.547746 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547706 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca" (OuterVolumeSpecName: "service-ca") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:57.547871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547740 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config" (OuterVolumeSpecName: "console-config") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:57.547871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547750 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:57.547871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.547832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:34:57.549474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.549439 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz" (OuterVolumeSpecName: "kube-api-access-hh2fz") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "kube-api-access-hh2fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:34:57.549474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.549468 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:34:57.549612 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.549446 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b524afd6-00fc-4a78-b065-bd9bf027d732" (UID: "b524afd6-00fc-4a78-b065-bd9bf027d732"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:34:57.648118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648086 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648116 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-console-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648125 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-oauth-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648135 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b524afd6-00fc-4a78-b065-bd9bf027d732-trusted-ca-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648144 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-oauth-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648153 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2fz\" (UniqueName: \"kubernetes.io/projected/b524afd6-00fc-4a78-b065-bd9bf027d732-kube-api-access-hh2fz\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:57.648282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:57.648163 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b524afd6-00fc-4a78-b065-bd9bf027d732-console-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:34:58.158226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.158192 2559 generic.go:356] "Generic (PLEG): container finished" podID="b524afd6-00fc-4a78-b065-bd9bf027d732" containerID="a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4" exitCode=0 Sep 10 13:34:58.158627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.158232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b7b89-lbxc8" event={"ID":"b524afd6-00fc-4a78-b065-bd9bf027d732","Type":"ContainerDied","Data":"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4"} Sep 10 13:34:58.158627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.158242 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b7b89-lbxc8" Sep 10 13:34:58.158627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.158283 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b7b89-lbxc8" event={"ID":"b524afd6-00fc-4a78-b065-bd9bf027d732","Type":"ContainerDied","Data":"b27892f4b5577e9305ce25a21fba15d8a8b95127c383df9ccf7a930e59d5327f"} Sep 10 13:34:58.158627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.158306 2559 scope.go:117] "RemoveContainer" containerID="a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4" Sep 10 13:34:58.167935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.167917 2559 scope.go:117] "RemoveContainer" containerID="a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4" Sep 10 13:34:58.168240 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:34:58.168221 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4\": container with ID starting with a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4 not found: ID does not exist" containerID="a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4" Sep 10 13:34:58.168291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.168247 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4"} err="failed to get container status \"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4\": rpc error: code = NotFound desc = could not find container \"a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4\": container with ID starting with a1ea869a0dba1786a2a50ec6a9bbc2bc987f1ebd873469bd6f77dddb672099b4 not found: ID does not exist" Sep 10 13:34:58.194964 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.194943 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:34:58.201133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.201112 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9477b7b89-lbxc8"] Sep 10 13:34:58.916285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:34:58.916254 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b524afd6-00fc-4a78-b065-bd9bf027d732" path="/var/lib/kubelet/pods/b524afd6-00fc-4a78-b065-bd9bf027d732/volumes" Sep 10 13:35:02.181254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:02.181221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:03.528314 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.528266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk"] Sep 10 13:35:03.528872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.528766 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b524afd6-00fc-4a78-b065-bd9bf027d732" containerName="console" Sep 10 13:35:03.528872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.528789 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b524afd6-00fc-4a78-b065-bd9bf027d732" containerName="console" Sep 10 13:35:03.528993 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.528879 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b524afd6-00fc-4a78-b065-bd9bf027d732" containerName="console" Sep 10 13:35:03.533584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.533559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.536549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.536522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kptmg\"" Sep 10 13:35:03.537250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.537228 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:35:03.537308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.537228 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:35:03.552188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.552159 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk"] Sep 10 13:35:03.691334 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.691312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.691484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.691349 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzvj\" (UniqueName: \"kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.691484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.691430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.792600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.792533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.792600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.792573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzvj\" (UniqueName: \"kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.792600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.792597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.792906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.792888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.792944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.792916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.803997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.803970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzvj\" (UniqueName: \"kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:03.843758 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:03.843738 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:04.128128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:04.128049 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk"] Sep 10 13:35:04.132351 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:04.132323 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c354781_f49b_4a7c_8214_8eb30f91a4ee.slice/crio-27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b WatchSource:0}: Error finding container 27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b: Status 404 returned error can't find the container with id 27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b Sep 10 13:35:04.186277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:04.186251 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" event={"ID":"3c354781-f49b-4a7c-8214-8eb30f91a4ee","Type":"ContainerStarted","Data":"27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b"} Sep 10 13:35:05.850665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:05.850631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:11.218931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:11.218890 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerID="e859ea8b2868cab83acf612b93f8d3bae855ec96a68eacbff61f480dee8f17b4" exitCode=0 Sep 10 13:35:11.219335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:11.218988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" event={"ID":"3c354781-f49b-4a7c-8214-8eb30f91a4ee","Type":"ContainerDied","Data":"e859ea8b2868cab83acf612b93f8d3bae855ec96a68eacbff61f480dee8f17b4"} Sep 10 13:35:14.236766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:14.236730 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerID="3f739d6cb56767a0f907a783e4f51becfafcb921836aa4514b9e6b00ab239797" exitCode=0 Sep 10 13:35:14.237132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:14.236805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" event={"ID":"3c354781-f49b-4a7c-8214-8eb30f91a4ee","Type":"ContainerDied","Data":"3f739d6cb56767a0f907a783e4f51becfafcb921836aa4514b9e6b00ab239797"} Sep 10 13:35:14.487160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:14.487076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:21.271530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:21.271491 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerID="a9e9e68ce5bb4fdccf4a9f510888d5f2046d5b8148a67acb7c3a52ada2f3888e" exitCode=0 Sep 10 13:35:21.271919 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:21.271557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" event={"ID":"3c354781-f49b-4a7c-8214-8eb30f91a4ee","Type":"ContainerDied","Data":"a9e9e68ce5bb4fdccf4a9f510888d5f2046d5b8148a67acb7c3a52ada2f3888e"} Sep 10 13:35:22.403981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.403958 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:22.425113 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.425089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle\") pod \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " Sep 10 13:35:22.425204 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.425142 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util\") pod \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " Sep 10 13:35:22.425204 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.425166 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkzvj\" (UniqueName: \"kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj\") pod \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\" (UID: \"3c354781-f49b-4a7c-8214-8eb30f91a4ee\") " Sep 10 13:35:22.425712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.425682 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle" (OuterVolumeSpecName: "bundle") pod "3c354781-f49b-4a7c-8214-8eb30f91a4ee" (UID: "3c354781-f49b-4a7c-8214-8eb30f91a4ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:22.427306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.427272 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj" (OuterVolumeSpecName: "kube-api-access-nkzvj") pod "3c354781-f49b-4a7c-8214-8eb30f91a4ee" (UID: "3c354781-f49b-4a7c-8214-8eb30f91a4ee"). InnerVolumeSpecName "kube-api-access-nkzvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:22.430175 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.430145 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util" (OuterVolumeSpecName: "util") pod "3c354781-f49b-4a7c-8214-8eb30f91a4ee" (UID: "3c354781-f49b-4a7c-8214-8eb30f91a4ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:22.525817 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.525738 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:22.525817 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.525775 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c354781-f49b-4a7c-8214-8eb30f91a4ee-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:22.525817 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:22.525784 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzvj\" (UniqueName: \"kubernetes.io/projected/3c354781-f49b-4a7c-8214-8eb30f91a4ee-kube-api-access-nkzvj\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:23.281482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:23.281448 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" event={"ID":"3c354781-f49b-4a7c-8214-8eb30f91a4ee","Type":"ContainerDied","Data":"27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b"} Sep 10 13:35:23.281643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:23.281497 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c30614cb65dda7513156029467b7d8cc27b2faaa9741f1d64b6b722e25964b" Sep 10 13:35:23.281643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:23.281494 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w42gk" Sep 10 13:35:23.676062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:23.675966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:24.629808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:24.629777 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:24.810510 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:24.810482 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:35:24.811557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:24.811537 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:35:24.821880 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:24.821861 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 13:35:25.547161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:25.547128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:26.581140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581107 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p"] Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581414 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="extract" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581427 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="extract" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581443 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="util" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581448 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="util" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581454 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="pull" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581460 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="pull" Sep 10 13:35:26.581513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.581507 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c354781-f49b-4a7c-8214-8eb30f91a4ee" containerName="extract" Sep 10 13:35:26.584266 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.584249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.594796 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.594776 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-76wbn\"" Sep 10 13:35:26.594894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.594816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:35:26.595678 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.595660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 13:35:26.606074 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.606049 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p"] Sep 10 13:35:26.655667 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.655631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.655803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.655681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdm2\" (UniqueName: \"kubernetes.io/projected/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-kube-api-access-fjdm2\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.756085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.756054 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.756217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.756096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdm2\" (UniqueName: \"kubernetes.io/projected/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-kube-api-access-fjdm2\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.756526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.756506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.768420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.768399 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdm2\" (UniqueName: \"kubernetes.io/projected/b9251ce9-f6c1-4ec4-9bad-484f115c39e8-kube-api-access-fjdm2\") pod \"cert-manager-operator-controller-manager-55c444bf86-wl46p\" (UID: \"b9251ce9-f6c1-4ec4-9bad-484f115c39e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:26.892421 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:26.892330 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" Sep 10 13:35:27.203990 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:27.203954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p"] Sep 10 13:35:27.205853 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:27.205826 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9251ce9_f6c1_4ec4_9bad_484f115c39e8.slice/crio-8e562ccc0b76bd4b23dbf29080ac4ef689dc3eb8e7a159e4d55b94e7b094a4ab WatchSource:0}: Error finding container 8e562ccc0b76bd4b23dbf29080ac4ef689dc3eb8e7a159e4d55b94e7b094a4ab: Status 404 returned error can't find the container with id 8e562ccc0b76bd4b23dbf29080ac4ef689dc3eb8e7a159e4d55b94e7b094a4ab Sep 10 13:35:27.208826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:27.208805 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:35:27.297518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:27.297488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" event={"ID":"b9251ce9-f6c1-4ec4-9bad-484f115c39e8","Type":"ContainerStarted","Data":"8e562ccc0b76bd4b23dbf29080ac4ef689dc3eb8e7a159e4d55b94e7b094a4ab"} Sep 10 13:35:28.394520 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:28.394487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:28.492985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:28.492946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:29.308977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:29.308945 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" event={"ID":"b9251ce9-f6c1-4ec4-9bad-484f115c39e8","Type":"ContainerStarted","Data":"3ae46579ae59400374f0aaf3fefe706ddd7b1c280002e7e1b3f8b6e7c3bc4d7a"} Sep 10 13:35:29.338610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:29.338555 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wl46p" podStartSLOduration=1.371268133 podStartE2EDuration="3.338536874s" totalImagesPullingTime="1.967268741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:29.336739175 +0000 UTC m=+304.946524085" watchObservedRunningTime="2026-09-10 13:35:29.338536874 +0000 UTC m=+304.948321788" Sep 10 13:35:29.787284 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:29.787251 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:29.902451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:29.902411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:30.101410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.101312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:30.203193 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.203160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:30.473441 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.473349 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8"] Sep 10 13:35:30.476915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.476894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.480417 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.480397 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kptmg\"" Sep 10 13:35:30.480523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.480475 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:35:30.481362 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.481346 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:35:30.487054 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.487035 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8"] Sep 10 13:35:30.582384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.582345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.582514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.582394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqq4\" (UniqueName: \"kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.582514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.582423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.683188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.683153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.683188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.683188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqq4\" (UniqueName: \"kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.683361 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.683212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.683555 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.683536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.683616 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.683599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.696736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.696713 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqq4\" (UniqueName: \"kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:30.785954 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:30.785927 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:31.083796 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:31.083768 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde137305_8dee_4f58_a4cb_41bd23d460b4.slice/crio-68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d WatchSource:0}: Error finding container 68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d: Status 404 returned error can't find the container with id 68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d Sep 10 13:35:31.091746 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:31.091726 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8"] Sep 10 13:35:31.319938 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:31.319902 2559 generic.go:356] "Generic (PLEG): container finished" podID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerID="01808eb4a498b1daa1df8287bb7c2fe8b87064490d32a57212cd1c10a3611909" exitCode=0 Sep 10 13:35:31.320095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:31.319969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" event={"ID":"de137305-8dee-4f58-a4cb-41bd23d460b4","Type":"ContainerDied","Data":"01808eb4a498b1daa1df8287bb7c2fe8b87064490d32a57212cd1c10a3611909"} Sep 10 13:35:31.320095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:31.320017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" event={"ID":"de137305-8dee-4f58-a4cb-41bd23d460b4","Type":"ContainerStarted","Data":"68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d"} Sep 10 13:35:32.733672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:32.733632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:32.962943 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:32.962912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:34.339005 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:34.338969 2559 generic.go:356] "Generic (PLEG): container finished" podID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerID="b83ccec73f5e298bd731f53ab21ae2e0b6d16d3507ae5bd49eecfabde067b63f" exitCode=0 Sep 10 13:35:34.339421 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:34.339051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" event={"ID":"de137305-8dee-4f58-a4cb-41bd23d460b4","Type":"ContainerDied","Data":"b83ccec73f5e298bd731f53ab21ae2e0b6d16d3507ae5bd49eecfabde067b63f"} Sep 10 13:35:35.344800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:35.344760 2559 generic.go:356] "Generic (PLEG): container finished" podID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerID="2d7429106994070a962066ee02fd82e94490022784e0c0e129522c2dde3593c9" exitCode=0 Sep 10 13:35:35.344800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:35.344799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" event={"ID":"de137305-8dee-4f58-a4cb-41bd23d460b4","Type":"ContainerDied","Data":"2d7429106994070a962066ee02fd82e94490022784e0c0e129522c2dde3593c9"} Sep 10 13:35:35.577100 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:35.577059 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:35.680182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:35.680100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:36.472130 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.472105 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:36.532733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.532706 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util\") pod \"de137305-8dee-4f58-a4cb-41bd23d460b4\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " Sep 10 13:35:36.532847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.532754 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwqq4\" (UniqueName: \"kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4\") pod \"de137305-8dee-4f58-a4cb-41bd23d460b4\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " Sep 10 13:35:36.532847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.532770 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle\") pod \"de137305-8dee-4f58-a4cb-41bd23d460b4\" (UID: \"de137305-8dee-4f58-a4cb-41bd23d460b4\") " Sep 10 13:35:36.533194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.533165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle" (OuterVolumeSpecName: "bundle") pod "de137305-8dee-4f58-a4cb-41bd23d460b4" (UID: "de137305-8dee-4f58-a4cb-41bd23d460b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:36.534907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.534880 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4" (OuterVolumeSpecName: "kube-api-access-zwqq4") pod "de137305-8dee-4f58-a4cb-41bd23d460b4" (UID: "de137305-8dee-4f58-a4cb-41bd23d460b4"). InnerVolumeSpecName "kube-api-access-zwqq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:36.537052 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.537032 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util" (OuterVolumeSpecName: "util") pod "de137305-8dee-4f58-a4cb-41bd23d460b4" (UID: "de137305-8dee-4f58-a4cb-41bd23d460b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:36.634040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.633960 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqq4\" (UniqueName: \"kubernetes.io/projected/de137305-8dee-4f58-a4cb-41bd23d460b4-kube-api-access-zwqq4\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:36.634040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.633988 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:36.634040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:36.634001 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de137305-8dee-4f58-a4cb-41bd23d460b4-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:37.355595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:37.355558 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" event={"ID":"de137305-8dee-4f58-a4cb-41bd23d460b4","Type":"ContainerDied","Data":"68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d"} Sep 10 13:35:37.355595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:37.355595 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4slc8" Sep 10 13:35:37.355784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:37.355612 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ecc71baeedb6d6a259d670e130b6d8bd05e314fb9051d1253469941094273d" Sep 10 13:35:38.226163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:38.226097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:38.324119 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:38.324091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:39.923844 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:39.923812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:40.423051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423017 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jzbrs"] Sep 10 13:35:40.423336 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423324 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="pull" Sep 10 13:35:40.423397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423338 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="pull" Sep 10 13:35:40.423397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423346 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="extract" Sep 10 13:35:40.423397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423351 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="extract" Sep 10 13:35:40.423397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423380 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="util" Sep 10 13:35:40.423397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423388 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="util" Sep 10 13:35:40.423589 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.423441 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="de137305-8dee-4f58-a4cb-41bd23d460b4" containerName="extract" Sep 10 13:35:40.426319 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.426295 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.428755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.428737 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 13:35:40.428858 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.428741 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 13:35:40.429577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.429562 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5hld5\"" Sep 10 13:35:40.441561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.441543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jzbrs"] Sep 10 13:35:40.464189 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.464167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzfq\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-kube-api-access-rzzfq\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.464283 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.464210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-bound-sa-token\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.565317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.565291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzfq\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-kube-api-access-rzzfq\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.565455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.565343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-bound-sa-token\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.574746 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.574719 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-bound-sa-token\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.574833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.574785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzfq\" (UniqueName: \"kubernetes.io/projected/1344d715-b551-4d7b-932c-c869d31b6148-kube-api-access-rzzfq\") pod \"cert-manager-55b4d954d5-jzbrs\" (UID: \"1344d715-b551-4d7b-932c-c869d31b6148\") " pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:40.747332 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:40.747282 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jzbrs" Sep 10 13:35:41.033899 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:41.033872 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jzbrs"] Sep 10 13:35:41.036655 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:41.036624 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1344d715_b551_4d7b_932c_c869d31b6148.slice/crio-79ff12a0013777de9c17028bbab29e57e0576091c13725e83cd6760d5ec99841 WatchSource:0}: Error finding container 79ff12a0013777de9c17028bbab29e57e0576091c13725e83cd6760d5ec99841: Status 404 returned error can't find the container with id 79ff12a0013777de9c17028bbab29e57e0576091c13725e83cd6760d5ec99841 Sep 10 13:35:41.374049 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:41.373978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jzbrs" event={"ID":"1344d715-b551-4d7b-932c-c869d31b6148","Type":"ContainerStarted","Data":"79ff12a0013777de9c17028bbab29e57e0576091c13725e83cd6760d5ec99841"} Sep 10 13:35:42.657529 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:42.657487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:42.760656 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:42.760623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:44.029534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.029505 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk"] Sep 10 13:35:44.032593 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.032577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.034960 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.034941 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 13:35:44.035824 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.035811 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-w2frp\"" Sep 10 13:35:44.035879 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.035827 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 13:35:44.042937 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.042918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk"] Sep 10 13:35:44.090903 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.090883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f051725-362f-4eb1-b1b7-38facdfcfef2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.091007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.090951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdbv\" (UniqueName: \"kubernetes.io/projected/5f051725-362f-4eb1-b1b7-38facdfcfef2-kube-api-access-lwdbv\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.191944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.191885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdbv\" (UniqueName: \"kubernetes.io/projected/5f051725-362f-4eb1-b1b7-38facdfcfef2-kube-api-access-lwdbv\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.191944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.191926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f051725-362f-4eb1-b1b7-38facdfcfef2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.192268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.192247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f051725-362f-4eb1-b1b7-38facdfcfef2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.201263 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.201235 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdbv\" (UniqueName: \"kubernetes.io/projected/5f051725-362f-4eb1-b1b7-38facdfcfef2-kube-api-access-lwdbv\") pod \"openshift-lws-operator-fd8ccff4c-qb8vk\" (UID: \"5f051725-362f-4eb1-b1b7-38facdfcfef2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.341678 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.341653 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" Sep 10 13:35:44.403261 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.402907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jzbrs" event={"ID":"1344d715-b551-4d7b-932c-c869d31b6148","Type":"ContainerStarted","Data":"9cc94e0a1e2686d8b64d61f88d7d3b3cace4da8bce187bb961a232201b3c8381"} Sep 10 13:35:44.423425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.423381 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jzbrs" podStartSLOduration=1.544821474 podStartE2EDuration="4.423352805s" totalImagesPullingTime="2.878531331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:44.421685637 +0000 UTC m=+320.031470549" watchObservedRunningTime="2026-09-10 13:35:44.423352805 +0000 UTC m=+320.033137719" Sep 10 13:35:44.615736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.615710 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk"] Sep 10 13:35:44.617934 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:44.617899 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f051725_362f_4eb1_b1b7_38facdfcfef2.slice/crio-35c833730cde11e5c0c735765df1ae39bc6e1f2062785915e406e71e2ab99a6f WatchSource:0}: Error finding container 35c833730cde11e5c0c735765df1ae39bc6e1f2062785915e406e71e2ab99a6f: Status 404 returned error can't find the container with id 35c833730cde11e5c0c735765df1ae39bc6e1f2062785915e406e71e2ab99a6f Sep 10 13:35:44.816337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.816310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:44.999910 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:44.999842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:45.408505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:45.408465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" event={"ID":"5f051725-362f-4eb1-b1b7-38facdfcfef2","Type":"ContainerStarted","Data":"35c833730cde11e5c0c735765df1ae39bc6e1f2062785915e406e71e2ab99a6f"} Sep 10 13:35:50.438902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:50.438864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" event={"ID":"5f051725-362f-4eb1-b1b7-38facdfcfef2","Type":"ContainerStarted","Data":"954244d51375f433453500672bf16cc091988dfc288f243b266314700a584015"} Sep 10 13:35:50.457882 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:50.457829 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qb8vk" podStartSLOduration=1.230007664 podStartE2EDuration="6.45781507s" totalImagesPullingTime="5.227807406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:50.456016928 +0000 UTC m=+326.065801842" watchObservedRunningTime="2026-09-10 13:35:50.45781507 +0000 UTC m=+326.067599983" Sep 10 13:35:51.158101 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:51.158066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:51.262435 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:51.262397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:52.832704 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:52.832668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:35:53.050770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.050734 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj"] Sep 10 13:35:53.054439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.054415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.056823 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.056799 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kptmg\"" Sep 10 13:35:53.056977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.056957 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:35:53.057832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.057814 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:35:53.066547 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.066526 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj"] Sep 10 13:35:53.158636 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.158567 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.158759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.158636 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.158759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.158662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9m46\" (UniqueName: \"kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.259643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.259617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.259751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.259649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9m46\" (UniqueName: \"kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.259751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.259677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.259971 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.259953 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.260014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.259998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.270016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.269994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9m46\" (UniqueName: \"kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.364324 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.364305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:53.639595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:53.639559 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj"] Sep 10 13:35:53.641805 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:35:53.641766 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9180fe5_e0aa_432d_9b6e_69523afa434e.slice/crio-f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f WatchSource:0}: Error finding container f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f: Status 404 returned error can't find the container with id f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f Sep 10 13:35:54.459094 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:54.459059 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerID="c5f5753ff5856d8fc08b9893494d234fce5b8c1bd584911c6d72e3672dabd7cd" exitCode=0 Sep 10 13:35:54.459502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:54.459150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" event={"ID":"b9180fe5-e0aa-432d-9b6e-69523afa434e","Type":"ContainerDied","Data":"c5f5753ff5856d8fc08b9893494d234fce5b8c1bd584911c6d72e3672dabd7cd"} Sep 10 13:35:54.459502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:54.459211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" event={"ID":"b9180fe5-e0aa-432d-9b6e-69523afa434e","Type":"ContainerStarted","Data":"f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f"} Sep 10 13:35:56.468978 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:56.468936 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerID="e7e1c7e10eacdb9ce94d812a594e9199c08176f00c69ff184f7a52e43fbd6962" exitCode=0 Sep 10 13:35:56.469405 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:56.469018 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" event={"ID":"b9180fe5-e0aa-432d-9b6e-69523afa434e","Type":"ContainerDied","Data":"e7e1c7e10eacdb9ce94d812a594e9199c08176f00c69ff184f7a52e43fbd6962"} Sep 10 13:35:57.475997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:57.475959 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerID="f81bc133c10bd8346584d743e62934fbf86e9efd18d6924fe401f4ddc8c6921e" exitCode=0 Sep 10 13:35:57.476381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:57.476064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" event={"ID":"b9180fe5-e0aa-432d-9b6e-69523afa434e","Type":"ContainerDied","Data":"f81bc133c10bd8346584d743e62934fbf86e9efd18d6924fe401f4ddc8c6921e"} Sep 10 13:35:58.608963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.608934 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:58.701611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.701570 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9m46\" (UniqueName: \"kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46\") pod \"b9180fe5-e0aa-432d-9b6e-69523afa434e\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " Sep 10 13:35:58.701741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.701639 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle\") pod \"b9180fe5-e0aa-432d-9b6e-69523afa434e\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " Sep 10 13:35:58.701741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.701655 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util\") pod \"b9180fe5-e0aa-432d-9b6e-69523afa434e\" (UID: \"b9180fe5-e0aa-432d-9b6e-69523afa434e\") " Sep 10 13:35:58.702345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.702311 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle" (OuterVolumeSpecName: "bundle") pod "b9180fe5-e0aa-432d-9b6e-69523afa434e" (UID: "b9180fe5-e0aa-432d-9b6e-69523afa434e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:58.703660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.703639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46" (OuterVolumeSpecName: "kube-api-access-p9m46") pod "b9180fe5-e0aa-432d-9b6e-69523afa434e" (UID: "b9180fe5-e0aa-432d-9b6e-69523afa434e"). InnerVolumeSpecName "kube-api-access-p9m46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:35:58.709773 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.709734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util" (OuterVolumeSpecName: "util") pod "b9180fe5-e0aa-432d-9b6e-69523afa434e" (UID: "b9180fe5-e0aa-432d-9b6e-69523afa434e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:35:58.802533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.802503 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:58.802533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.802531 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9180fe5-e0aa-432d-9b6e-69523afa434e-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:58.802659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:58.802545 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9m46\" (UniqueName: \"kubernetes.io/projected/b9180fe5-e0aa-432d-9b6e-69523afa434e-kube-api-access-p9m46\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:35:59.488634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:59.488602 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" Sep 10 13:35:59.488634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:59.488610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2w8bj" event={"ID":"b9180fe5-e0aa-432d-9b6e-69523afa434e","Type":"ContainerDied","Data":"f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f"} Sep 10 13:35:59.488846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:35:59.488669 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f151a4009c6ddcc8a21981e7d3f88a5975d214f6a035d354692dbbffaf532a0f" Sep 10 13:36:06.633574 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633529 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq"] Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633853 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="util" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633867 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="util" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633884 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="pull" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633890 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="pull" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633908 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="extract" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633925 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="extract" Sep 10 13:36:06.634081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.633979 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9180fe5-e0aa-432d-9b6e-69523afa434e" containerName="extract" Sep 10 13:36:06.638626 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.638591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.641711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.641686 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:36:06.642260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.642236 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kptmg\"" Sep 10 13:36:06.642833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.642813 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:36:06.657519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.657488 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq"] Sep 10 13:36:06.761484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.761460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.761611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.761512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.761611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.761577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfp4\" (UniqueName: \"kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.861961 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.861927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.862064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.861981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfp4\" (UniqueName: \"kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.862064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.862028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.862309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.862288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.862431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.862411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.870823 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.870797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfp4\" (UniqueName: \"kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:06.948649 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:06.948586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:07.226684 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.226659 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq"] Sep 10 13:36:07.228800 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:07.228774 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c63102e_a846_4368_b957_e3edfeb6599f.slice/crio-7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313 WatchSource:0}: Error finding container 7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313: Status 404 returned error can't find the container with id 7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313 Sep 10 13:36:07.528591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.528566 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c63102e-a846-4368-b957-e3edfeb6599f" containerID="f407bb12730360e58d4af352b6ce7057324a49822c1f89315018b15da4ce270b" exitCode=0 Sep 10 13:36:07.528716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.528599 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" event={"ID":"7c63102e-a846-4368-b957-e3edfeb6599f","Type":"ContainerDied","Data":"f407bb12730360e58d4af352b6ce7057324a49822c1f89315018b15da4ce270b"} Sep 10 13:36:07.528716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.528634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" event={"ID":"7c63102e-a846-4368-b957-e3edfeb6599f","Type":"ContainerStarted","Data":"7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313"} Sep 10 13:36:07.715414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.715389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:07.806724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:07.806667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:08.752268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.752232 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms"] Sep 10 13:36:08.756672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.756648 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.759800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.759778 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 13:36:08.760239 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.760215 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 13:36:08.761195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.761174 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sf2hc\"" Sep 10 13:36:08.762751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.762727 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 13:36:08.762885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.762870 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 13:36:08.778039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.778014 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms"] Sep 10 13:36:08.878007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.877984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvlfj\" (UniqueName: \"kubernetes.io/projected/7da1dc22-e1bd-4774-941c-04f3e4f85464-kube-api-access-mvlfj\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.878109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.878013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.878109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.878038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.878109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.878062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.878109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.878105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.978584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.978562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.978684 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.978594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.978684 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.978642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlfj\" (UniqueName: \"kubernetes.io/projected/7da1dc22-e1bd-4774-941c-04f3e4f85464-kube-api-access-mvlfj\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.978684 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.978659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.978824 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.978695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.979124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.979104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-manifests\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.980832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.980804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7da1dc22-e1bd-4774-941c-04f3e4f85464-tmp\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.980937 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.980855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-apiservice-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.980999 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.980936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7da1dc22-e1bd-4774-941c-04f3e4f85464-webhook-cert\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.988549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.988526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlfj\" (UniqueName: \"kubernetes.io/projected/7da1dc22-e1bd-4774-941c-04f3e4f85464-kube-api-access-mvlfj\") pod \"opendatahub-operator-controller-manager-94d598c87-gqbms\" (UID: \"7da1dc22-e1bd-4774-941c-04f3e4f85464\") " pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:08.989737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:08.989711 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:09.068311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.068289 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:09.084991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.084962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:09.185647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.185622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:09.287408 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.287345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:09.379203 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:09.379178 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da1dc22_e1bd_4774_941c_04f3e4f85464.slice/crio-fd18587b48ffe246e3668a0467e5b6f8176ec57d4c33729434a105357f279d7a WatchSource:0}: Error finding container fd18587b48ffe246e3668a0467e5b6f8176ec57d4c33729434a105357f279d7a: Status 404 returned error can't find the container with id fd18587b48ffe246e3668a0467e5b6f8176ec57d4c33729434a105357f279d7a Sep 10 13:36:09.380272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.380169 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms"] Sep 10 13:36:09.539813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.539778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" event={"ID":"7da1dc22-e1bd-4774-941c-04f3e4f85464","Type":"ContainerStarted","Data":"fd18587b48ffe246e3668a0467e5b6f8176ec57d4c33729434a105357f279d7a"} Sep 10 13:36:09.541556 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.541532 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c63102e-a846-4368-b957-e3edfeb6599f" containerID="d2598e5d6b4401e777fb43b1135089826c47d3c1fa876c9fa385887a09bdf081" exitCode=0 Sep 10 13:36:09.541647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:09.541579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" event={"ID":"7c63102e-a846-4368-b957-e3edfeb6599f","Type":"ContainerDied","Data":"d2598e5d6b4401e777fb43b1135089826c47d3c1fa876c9fa385887a09bdf081"} Sep 10 13:36:10.551213 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:10.551180 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c63102e-a846-4368-b957-e3edfeb6599f" containerID="490c2b78d0d9e2fac65f426b18197894bc5050269387b98584c2acca39bc0e9a" exitCode=0 Sep 10 13:36:10.551749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:10.551241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" event={"ID":"7c63102e-a846-4368-b957-e3edfeb6599f","Type":"ContainerDied","Data":"490c2b78d0d9e2fac65f426b18197894bc5050269387b98584c2acca39bc0e9a"} Sep 10 13:36:12.849554 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:12.849529 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:13.012559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.012531 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brfp4\" (UniqueName: \"kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4\") pod \"7c63102e-a846-4368-b957-e3edfeb6599f\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " Sep 10 13:36:13.012724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.012708 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle\") pod \"7c63102e-a846-4368-b957-e3edfeb6599f\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " Sep 10 13:36:13.012792 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.012767 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util\") pod \"7c63102e-a846-4368-b957-e3edfeb6599f\" (UID: \"7c63102e-a846-4368-b957-e3edfeb6599f\") " Sep 10 13:36:13.013477 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.013452 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle" (OuterVolumeSpecName: "bundle") pod "7c63102e-a846-4368-b957-e3edfeb6599f" (UID: "7c63102e-a846-4368-b957-e3edfeb6599f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:13.014443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.014415 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4" (OuterVolumeSpecName: "kube-api-access-brfp4") pod "7c63102e-a846-4368-b957-e3edfeb6599f" (UID: "7c63102e-a846-4368-b957-e3edfeb6599f"). InnerVolumeSpecName "kube-api-access-brfp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:36:13.017149 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.017126 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util" (OuterVolumeSpecName: "util") pod "7c63102e-a846-4368-b957-e3edfeb6599f" (UID: "7c63102e-a846-4368-b957-e3edfeb6599f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:13.113249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.113197 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:13.113249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.113217 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c63102e-a846-4368-b957-e3edfeb6599f-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:13.113249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.113226 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfp4\" (UniqueName: \"kubernetes.io/projected/7c63102e-a846-4368-b957-e3edfeb6599f-kube-api-access-brfp4\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:13.567854 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.567826 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" Sep 10 13:36:13.568016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.567822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0p9qcq" event={"ID":"7c63102e-a846-4368-b957-e3edfeb6599f","Type":"ContainerDied","Data":"7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313"} Sep 10 13:36:13.568016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.567919 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7232439d8763b30714cd84b1178f12343cd2b9ab4748e9f0bf9c8239a3900313" Sep 10 13:36:13.569644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:13.569608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" event={"ID":"7da1dc22-e1bd-4774-941c-04f3e4f85464","Type":"ContainerStarted","Data":"e6c11d7d83df51fe9821bfe36d7cc14238b7def25cec07b3644af032f2e35fec"} Sep 10 13:36:14.575742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:14.575709 2559 generic.go:356] "Generic (PLEG): container finished" podID="7da1dc22-e1bd-4774-941c-04f3e4f85464" containerID="e6c11d7d83df51fe9821bfe36d7cc14238b7def25cec07b3644af032f2e35fec" exitCode=0 Sep 10 13:36:14.576152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:14.575797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" event={"ID":"7da1dc22-e1bd-4774-941c-04f3e4f85464","Type":"ContainerDied","Data":"e6c11d7d83df51fe9821bfe36d7cc14238b7def25cec07b3644af032f2e35fec"} Sep 10 13:36:15.587715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:15.587672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" event={"ID":"7da1dc22-e1bd-4774-941c-04f3e4f85464","Type":"ContainerStarted","Data":"662a8ecbf1221672e0d2519ee94310de6d1d30203ebe86aa99aadd1cff00d26a"} Sep 10 13:36:15.588055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:15.587854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:15.619310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:15.619263 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" podStartSLOduration=2.671372183 podStartE2EDuration="7.619249415s" totalImagesPullingTime="3.871449209s" totalInitContainerRuntime="1.076428023s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:15.616353517 +0000 UTC m=+351.226138430" watchObservedRunningTime="2026-09-10 13:36:15.619249415 +0000 UTC m=+351.229034327" Sep 10 13:36:17.546885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:17.546812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:17.634878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:17.634843 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:26.594798 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:26.594762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-94d598c87-gqbms" Sep 10 13:36:29.246548 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:29.246508 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:29.387741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:29.387703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:32.512396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.512351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:32.545161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545135 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc"] Sep 10 13:36:32.545450 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545438 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="pull" Sep 10 13:36:32.545505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545451 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="pull" Sep 10 13:36:32.545505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545465 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="util" Sep 10 13:36:32.545505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545471 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="util" Sep 10 13:36:32.545505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545492 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="extract" Sep 10 13:36:32.545505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545497 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="extract" Sep 10 13:36:32.545655 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.545551 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c63102e-a846-4368-b957-e3edfeb6599f" containerName="extract" Sep 10 13:36:32.554099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.554075 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.557832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.557811 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nhnw5\"" Sep 10 13:36:32.557832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.557826 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 13:36:32.557996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.557854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 13:36:32.572475 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.572454 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc"] Sep 10 13:36:32.595725 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.595696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:32.654957 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.654930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f50a90b-6907-4185-9fe1-26661d8f2f86-tls-certs\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.655059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.654987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlcm\" (UniqueName: \"kubernetes.io/projected/8f50a90b-6907-4185-9fe1-26661d8f2f86-kube-api-access-4rlcm\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.655059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.655013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f50a90b-6907-4185-9fe1-26661d8f2f86-tmp\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.756173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.756148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlcm\" (UniqueName: \"kubernetes.io/projected/8f50a90b-6907-4185-9fe1-26661d8f2f86-kube-api-access-4rlcm\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.756304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.756183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f50a90b-6907-4185-9fe1-26661d8f2f86-tmp\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.756304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.756215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f50a90b-6907-4185-9fe1-26661d8f2f86-tls-certs\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.758538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.758514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f50a90b-6907-4185-9fe1-26661d8f2f86-tmp\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.758675 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.758658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f50a90b-6907-4185-9fe1-26661d8f2f86-tls-certs\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.768588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.768539 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlcm\" (UniqueName: \"kubernetes.io/projected/8f50a90b-6907-4185-9fe1-26661d8f2f86-kube-api-access-4rlcm\") pod \"kube-auth-proxy-66cb58b9b9-xjgxc\" (UID: \"8f50a90b-6907-4185-9fe1-26661d8f2f86\") " pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:32.863668 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:32.863648 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" Sep 10 13:36:33.159278 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:33.159212 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc"] Sep 10 13:36:33.162096 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:33.162065 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f50a90b_6907_4185_9fe1_26661d8f2f86.slice/crio-0468b973513fb56751fc7157282eb4f718f7b2c3a178487c20923966d2d988c2 WatchSource:0}: Error finding container 0468b973513fb56751fc7157282eb4f718f7b2c3a178487c20923966d2d988c2: Status 404 returned error can't find the container with id 0468b973513fb56751fc7157282eb4f718f7b2c3a178487c20923966d2d988c2 Sep 10 13:36:33.673951 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:33.673925 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" event={"ID":"8f50a90b-6907-4185-9fe1-26661d8f2f86","Type":"ContainerStarted","Data":"0468b973513fb56751fc7157282eb4f718f7b2c3a178487c20923966d2d988c2"} Sep 10 13:36:34.175803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:34.175766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:34.252376 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:34.252339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:35.631964 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:35.631922 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:36.691165 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:36.691129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" event={"ID":"8f50a90b-6907-4185-9fe1-26661d8f2f86","Type":"ContainerStarted","Data":"910a16225001af3ccb71066a1d2f4c307a94021a167d9f218c5401540aafac4b"} Sep 10 13:36:36.709518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:36.709471 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66cb58b9b9-xjgxc" podStartSLOduration=1.953451608 podStartE2EDuration="4.709456763s" totalImagesPullingTime="2.756005155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:36.708617676 +0000 UTC m=+372.318402589" watchObservedRunningTime="2026-09-10 13:36:36.709456763 +0000 UTC m=+372.319241675" Sep 10 13:36:39.269161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.269110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:39.370922 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.370882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:39.522237 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.522148 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:39.526113 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.526089 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.529066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.529037 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 13:36:39.529066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.529057 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hc8pz\"" Sep 10 13:36:39.543697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.543672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:39.561583 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.561554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:39.711000 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.710973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711069 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711152 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bzb\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.711317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.711228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.748004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.747974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:39.812140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bzb\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812667 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812754 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812819 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.812914 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.812895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.813182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.813161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.814712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.814690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.814800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.814722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.823194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.823167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bzb\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.823604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.823583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.836476 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.836456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:39.851605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:39.851582 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:40.115027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.115001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:40.116876 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:40.116851 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf361290_5a9c_4773_8b38_7f0523d6eba9.slice/crio-3bfb8590f35f30394ec52ceefcdfecc45f70eb50b016638bc7527aa542c2da6b WatchSource:0}: Error finding container 3bfb8590f35f30394ec52ceefcdfecc45f70eb50b016638bc7527aa542c2da6b: Status 404 returned error can't find the container with id 3bfb8590f35f30394ec52ceefcdfecc45f70eb50b016638bc7527aa542c2da6b Sep 10 13:36:40.179462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.179430 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-88fwh"] Sep 10 13:36:40.184520 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.184499 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.188242 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.188222 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 13:36:40.188495 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.188466 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kxc4v\"" Sep 10 13:36:40.191567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.191546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-88fwh"] Sep 10 13:36:40.206304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.206267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:40.300012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.299982 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:40.317325 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.317295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-cert\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.317511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.317332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxcs\" (UniqueName: \"kubernetes.io/projected/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-kube-api-access-xcxcs\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.400785 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.400718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:40.418691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.418667 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-cert\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.418782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.418696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxcs\" (UniqueName: \"kubernetes.io/projected/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-kube-api-access-xcxcs\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.421016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.420997 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-cert\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.427165 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.427144 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxcs\" (UniqueName: \"kubernetes.io/projected/82925dd0-59f5-4df3-8c5c-6c0cb01126fb-kube-api-access-xcxcs\") pod \"odh-model-controller-7998c67dc4-88fwh\" (UID: \"82925dd0-59f5-4df3-8c5c-6c0cb01126fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.498557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.498536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:40.712382 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.712298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" event={"ID":"af361290-5a9c-4773-8b38-7f0523d6eba9","Type":"ContainerStarted","Data":"3bfb8590f35f30394ec52ceefcdfecc45f70eb50b016638bc7527aa542c2da6b"} Sep 10 13:36:40.771247 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.771225 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-88fwh"] Sep 10 13:36:40.773185 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:40.773156 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82925dd0_59f5_4df3_8c5c_6c0cb01126fb.slice/crio-e04948e65269f3652e672f74c4b3550daad88ae0daab1f81654231c31772c7cc WatchSource:0}: Error finding container e04948e65269f3652e672f74c4b3550daad88ae0daab1f81654231c31772c7cc: Status 404 returned error can't find the container with id e04948e65269f3652e672f74c4b3550daad88ae0daab1f81654231c31772c7cc Sep 10 13:36:40.922951 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:40.922910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:41.000993 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:41.000925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:41.720703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:41.720669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" event={"ID":"82925dd0-59f5-4df3-8c5c-6c0cb01126fb","Type":"ContainerStarted","Data":"e04948e65269f3652e672f74c4b3550daad88ae0daab1f81654231c31772c7cc"} Sep 10 13:36:44.524661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.524617 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:36:44.524958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.524704 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:36:44.524958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.524738 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:36:44.740786 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.740750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" event={"ID":"af361290-5a9c-4773-8b38-7f0523d6eba9","Type":"ContainerStarted","Data":"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828"} Sep 10 13:36:44.742464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.742437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" event={"ID":"82925dd0-59f5-4df3-8c5c-6c0cb01126fb","Type":"ContainerStarted","Data":"a84ff5d292a8f5b95bad656394d0935c39320c9f439a8636d26b2ad2a6fef39f"} Sep 10 13:36:44.742595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.742488 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:36:44.771136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.771094 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" podStartSLOduration=1.366478374 podStartE2EDuration="5.771080197s" totalImagesPullingTime="4.404601823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:44.765714091 +0000 UTC m=+380.375499004" watchObservedRunningTime="2026-09-10 13:36:44.771080197 +0000 UTC m=+380.380865110" Sep 10 13:36:44.798120 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.798042 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" podStartSLOduration=1.006054699 podStartE2EDuration="4.798024687s" totalImagesPullingTime="3.791969988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:44.795607372 +0000 UTC m=+380.405392284" watchObservedRunningTime="2026-09-10 13:36:44.798024687 +0000 UTC m=+380.407809601" Sep 10 13:36:44.836633 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:44.836608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:45.159916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:45.159840 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:45.838583 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:45.838551 2559 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 13:36:45.838927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:45.838609 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 13:36:46.751914 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:46.751874 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerName="istio-proxy" containerID="cri-o://4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828" gracePeriod=30 Sep 10 13:36:47.341807 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.341775 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m"] Sep 10 13:36:47.344953 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.344932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.347733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.347704 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-84mmp\"" Sep 10 13:36:47.347985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.347955 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 13:36:47.348093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.348069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 13:36:47.348538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.348525 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 13:36:47.362759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.362737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m"] Sep 10 13:36:47.371041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371016 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0777e25d-1995-4f94-b993-b93c755b9b15-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371171 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.371293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.371236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7kq\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-kube-api-access-zs7kq\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7kq\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-kube-api-access-zs7kq\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.472633 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.472430 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0777e25d-1995-4f94-b993-b93c755b9b15-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.473104 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.473081 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.474736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.474712 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.474841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.474801 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.475045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.475025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0777e25d-1995-4f94-b993-b93c755b9b15-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.475162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.475145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0777e25d-1995-4f94-b993-b93c755b9b15-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.481719 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.481690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.482029 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.482011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7kq\" (UniqueName: \"kubernetes.io/projected/0777e25d-1995-4f94-b993-b93c755b9b15-kube-api-access-zs7kq\") pod \"istiod-openshift-gateway-5677ccb49b-nnn7m\" (UID: \"0777e25d-1995-4f94-b993-b93c755b9b15\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.654752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.654677 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:47.933700 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:47.933653 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m"] Sep 10 13:36:47.936570 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:36:47.936538 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0777e25d_1995_4f94_b993_b93c755b9b15.slice/crio-bee6e3b76aedc8a77dad65ad1b63466db3c3cb8d9237f30dc4b2a446656b3098 WatchSource:0}: Error finding container bee6e3b76aedc8a77dad65ad1b63466db3c3cb8d9237f30dc4b2a446656b3098: Status 404 returned error can't find the container with id bee6e3b76aedc8a77dad65ad1b63466db3c3cb8d9237f30dc4b2a446656b3098 Sep 10 13:36:48.763946 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:48.763907 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" event={"ID":"0777e25d-1995-4f94-b993-b93c755b9b15","Type":"ContainerStarted","Data":"bee6e3b76aedc8a77dad65ad1b63466db3c3cb8d9237f30dc4b2a446656b3098"} Sep 10 13:36:50.676141 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:50.676053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:50.845851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:50.845756 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:50.943465 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:50.943354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:51.268644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:51.268604 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:36:51.268767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:51.268672 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:36:51.413081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:51.413042 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:51.783210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:51.783180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" event={"ID":"0777e25d-1995-4f94-b993-b93c755b9b15","Type":"ContainerStarted","Data":"f876877f1fde16c20c51ea977378a71139a7233c8fe3b92e1b348d4817c90e7f"} Sep 10 13:36:51.783585 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:51.783352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:52.005184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.005162 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:52.017663 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017604 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4bzb\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.017663 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017643 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.017829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017681 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.017829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.017829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017738 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.017829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017762 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.018035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017832 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.018035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017873 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.018035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017917 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy\") pod \"af361290-5a9c-4773-8b38-7f0523d6eba9\" (UID: \"af361290-5a9c-4773-8b38-7f0523d6eba9\") " Sep 10 13:36:52.018035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.017999 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:52.018231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018025 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:52.018231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data" (OuterVolumeSpecName: "istio-data") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:52.018231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018084 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:36:52.018434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:52.018434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018318 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.018434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018338 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af361290-5a9c-4773-8b38-7f0523d6eba9-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.018434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018347 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.018434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.018358 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.019976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.019947 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb" (OuterVolumeSpecName: "kube-api-access-x4bzb") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "kube-api-access-x4bzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:36:52.020107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.020085 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:36:52.020172 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.020101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:36:52.020231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.020175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token" (OuterVolumeSpecName: "istio-token") pod "af361290-5a9c-4773-8b38-7f0523d6eba9" (UID: "af361290-5a9c-4773-8b38-7f0523d6eba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:36:52.035653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.035601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" podStartSLOduration=1.706380813 podStartE2EDuration="5.035583076s" totalImagesPullingTime="3.329202263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:51.831884908 +0000 UTC m=+387.441669820" watchObservedRunningTime="2026-09-10 13:36:52.035583076 +0000 UTC m=+387.645367990" Sep 10 13:36:52.046192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.046114 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:52.119605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.119577 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bzb\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-kube-api-access-x4bzb\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.119605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.119602 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.119766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.119612 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.119766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.119622 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.119766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.119631 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af361290-5a9c-4773-8b38-7f0523d6eba9-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:36:52.133720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.133690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:52.231994 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.231966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:36:52.789870 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.789835 2559 generic.go:356] "Generic (PLEG): container finished" podID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerID="4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828" exitCode=0 Sep 10 13:36:52.790293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.789888 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" Sep 10 13:36:52.790293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.789922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" event={"ID":"af361290-5a9c-4773-8b38-7f0523d6eba9","Type":"ContainerDied","Data":"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828"} Sep 10 13:36:52.790293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.789996 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4" event={"ID":"af361290-5a9c-4773-8b38-7f0523d6eba9","Type":"ContainerDied","Data":"3bfb8590f35f30394ec52ceefcdfecc45f70eb50b016638bc7527aa542c2da6b"} Sep 10 13:36:52.790293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.790025 2559 scope.go:117] "RemoveContainer" containerID="4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828" Sep 10 13:36:52.792469 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.792445 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nnn7m" Sep 10 13:36:52.799884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.799867 2559 scope.go:117] "RemoveContainer" containerID="4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828" Sep 10 13:36:52.800157 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:36:52.800138 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828\": container with ID starting with 4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828 not found: ID does not exist" containerID="4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828" Sep 10 13:36:52.800235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.800166 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828"} err="failed to get container status \"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828\": rpc error: code = NotFound desc = could not find container \"4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828\": container with ID starting with 4158eaa078eadb56a1ef10795776fb53b73150ee289cf3a9c2b915395952d828 not found: ID does not exist" Sep 10 13:36:52.844802 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.844408 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:52.847855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.847826 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-4xjp4"] Sep 10 13:36:52.918302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:52.918266 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" path="/var/lib/kubelet/pods/af361290-5a9c-4773-8b38-7f0523d6eba9/volumes" Sep 10 13:36:55.749258 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:36:55.749225 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-88fwh" Sep 10 13:37:41.381893 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:41.381861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:41.528605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:41.528573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:45.780388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.780346 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm"] Sep 10 13:37:45.780759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.780686 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerName="istio-proxy" Sep 10 13:37:45.780759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.780699 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerName="istio-proxy" Sep 10 13:37:45.780759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.780748 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af361290-5a9c-4773-8b38-7f0523d6eba9" containerName="istio-proxy" Sep 10 13:37:45.788536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.788515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:45.794484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.794462 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 13:37:45.794659 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.794597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 13:37:45.794744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.794663 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kptmg\"" Sep 10 13:37:45.796895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.796868 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm"] Sep 10 13:37:45.889821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.889790 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9"] Sep 10 13:37:45.893540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.893523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:45.904939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.904919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9"] Sep 10 13:37:45.921578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.921553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrcv\" (UniqueName: \"kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:45.921676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.921593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:45.921676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.921625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:45.955624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.955600 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl"] Sep 10 13:37:45.961497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.960415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:45.969902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:45.969868 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl"] Sep 10 13:37:46.023013 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.022983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kdz\" (UniqueName: \"kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.023122 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.023122 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.023122 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.023279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrcv\" (UniqueName: \"kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.023279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.023388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.023438 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.023360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.039321 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.039276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrcv\" (UniqueName: \"kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.073686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.073663 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2"] Sep 10 13:37:46.077416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.077402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.097477 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.097458 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2"] Sep 10 13:37:46.098649 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.098630 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:46.124136 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.124236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.124236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6tx\" (UniqueName: \"kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.124236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kdz\" (UniqueName: \"kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.124396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.124396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.124607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.124642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.124619 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.140146 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.140124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kdz\" (UniqueName: \"kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.201776 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.201748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:46.225451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6tx\" (UniqueName: \"kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.225551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.225595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.225634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.225706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.225744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225719 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmm6\" (UniqueName: \"kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.225853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.225945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.225927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.236203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.236169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6tx\" (UniqueName: \"kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.272435 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.272413 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:46.327400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.327328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.327589 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.327557 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.327672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.327592 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmm6\" (UniqueName: \"kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.328126 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.328067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.328221 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.328191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.339396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.339357 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmm6\" (UniqueName: \"kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.386334 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.386307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:46.392090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.392072 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm"] Sep 10 13:37:46.393472 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:37:46.393441 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43109b54_782b_49b6_92b8_038b5147f18d.slice/crio-4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3 WatchSource:0}: Error finding container 4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3: Status 404 returned error can't find the container with id 4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3 Sep 10 13:37:46.479078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.479053 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9"] Sep 10 13:37:46.480471 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:37:46.480438 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf424e499_9a8e_4ac5_a4f6_eff65f0851f1.slice/crio-01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d WatchSource:0}: Error finding container 01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d: Status 404 returned error can't find the container with id 01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d Sep 10 13:37:46.556948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.555684 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl"] Sep 10 13:37:46.572150 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:37:46.572121 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59416be3_865b_4ed6_b1c2_07e62868f881.slice/crio-9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654 WatchSource:0}: Error finding container 9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654: Status 404 returned error can't find the container with id 9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654 Sep 10 13:37:46.672695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:46.672671 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2"] Sep 10 13:37:46.743152 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:37:46.743124 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd71ede_2461_411b_91e3_b7d06acf8650.slice/crio-26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9 WatchSource:0}: Error finding container 26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9: Status 404 returned error can't find the container with id 26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9 Sep 10 13:37:47.062667 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.062634 2559 generic.go:356] "Generic (PLEG): container finished" podID="59416be3-865b-4ed6-b1c2-07e62868f881" containerID="fbcf2c9d8363644e8b0fd87a245acdccf61e51f7e8f73db30df928409d3d525a" exitCode=0 Sep 10 13:37:47.063097 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.062676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" event={"ID":"59416be3-865b-4ed6-b1c2-07e62868f881","Type":"ContainerDied","Data":"fbcf2c9d8363644e8b0fd87a245acdccf61e51f7e8f73db30df928409d3d525a"} Sep 10 13:37:47.063097 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.062721 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" event={"ID":"59416be3-865b-4ed6-b1c2-07e62868f881","Type":"ContainerStarted","Data":"9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654"} Sep 10 13:37:47.064434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.064411 2559 generic.go:356] "Generic (PLEG): container finished" podID="43109b54-782b-49b6-92b8-038b5147f18d" containerID="90c536cc4937ceede1efd6b22fec161a4942fb5514620f8ab8e75a27b18f5de7" exitCode=0 Sep 10 13:37:47.064523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.064500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerDied","Data":"90c536cc4937ceede1efd6b22fec161a4942fb5514620f8ab8e75a27b18f5de7"} Sep 10 13:37:47.064572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.064548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerStarted","Data":"4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3"} Sep 10 13:37:47.066216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.066197 2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerID="7fa9e22f3d88fcba1410712b8acea2875662ff0472d433b4d44e6091986ffd9a" exitCode=0 Sep 10 13:37:47.066299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.066265 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" event={"ID":"fcd71ede-2461-411b-91e3-b7d06acf8650","Type":"ContainerDied","Data":"7fa9e22f3d88fcba1410712b8acea2875662ff0472d433b4d44e6091986ffd9a"} Sep 10 13:37:47.066355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.066298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" event={"ID":"fcd71ede-2461-411b-91e3-b7d06acf8650","Type":"ContainerStarted","Data":"26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9"} Sep 10 13:37:47.068012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.067978 2559 generic.go:356] "Generic (PLEG): container finished" podID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerID="b314b04891823f23b3ff5865063f0c13ef710892c1c6bccc8fa5a4bcf6a53abd" exitCode=0 Sep 10 13:37:47.068091 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.068003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" event={"ID":"f424e499-9a8e-4ac5-a4f6-eff65f0851f1","Type":"ContainerDied","Data":"b314b04891823f23b3ff5865063f0c13ef710892c1c6bccc8fa5a4bcf6a53abd"} Sep 10 13:37:47.068091 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.068048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" event={"ID":"f424e499-9a8e-4ac5-a4f6-eff65f0851f1","Type":"ContainerStarted","Data":"01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d"} Sep 10 13:37:47.576429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:47.576362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:48.076071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:48.076035 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerStarted","Data":"159ee1b72ab1069b07b860a80c8219bd049534ac43200b7089447661459bb8fa"} Sep 10 13:37:48.077816 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:48.077792 2559 generic.go:356] "Generic (PLEG): container finished" podID="59416be3-865b-4ed6-b1c2-07e62868f881" containerID="d760db6e448f091e80a7d7e77b6f8473529c59262a65570174fae66c3d2e6d6b" exitCode=0 Sep 10 13:37:48.077933 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:48.077867 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" event={"ID":"59416be3-865b-4ed6-b1c2-07e62868f881","Type":"ContainerDied","Data":"d760db6e448f091e80a7d7e77b6f8473529c59262a65570174fae66c3d2e6d6b"} Sep 10 13:37:49.084725 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.084689 2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerID="a31ae0df11eb43cae23343041f0f13803715e38aa061141cc29ca256f64c2af3" exitCode=0 Sep 10 13:37:49.085192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.084778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" event={"ID":"fcd71ede-2461-411b-91e3-b7d06acf8650","Type":"ContainerDied","Data":"a31ae0df11eb43cae23343041f0f13803715e38aa061141cc29ca256f64c2af3"} Sep 10 13:37:49.087157 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.087129 2559 generic.go:356] "Generic (PLEG): container finished" podID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerID="c96436005aec1c3b13c237ff5ae34392113adb69855f022068a08fb5cf092e6f" exitCode=0 Sep 10 13:37:49.087329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.087217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" event={"ID":"f424e499-9a8e-4ac5-a4f6-eff65f0851f1","Type":"ContainerDied","Data":"c96436005aec1c3b13c237ff5ae34392113adb69855f022068a08fb5cf092e6f"} Sep 10 13:37:49.090036 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.089983 2559 generic.go:356] "Generic (PLEG): container finished" podID="59416be3-865b-4ed6-b1c2-07e62868f881" containerID="253a52a4a9ec4fe8c4a708f64e7f1247424fa47a49a76c8ee0f7f8120e607aa0" exitCode=0 Sep 10 13:37:49.090121 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.090080 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" event={"ID":"59416be3-865b-4ed6-b1c2-07e62868f881","Type":"ContainerDied","Data":"253a52a4a9ec4fe8c4a708f64e7f1247424fa47a49a76c8ee0f7f8120e607aa0"} Sep 10 13:37:49.092106 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.092082 2559 generic.go:356] "Generic (PLEG): container finished" podID="43109b54-782b-49b6-92b8-038b5147f18d" containerID="159ee1b72ab1069b07b860a80c8219bd049534ac43200b7089447661459bb8fa" exitCode=0 Sep 10 13:37:49.092212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:49.092118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerDied","Data":"159ee1b72ab1069b07b860a80c8219bd049534ac43200b7089447661459bb8fa"} Sep 10 13:37:50.098999 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.098960 2559 generic.go:356] "Generic (PLEG): container finished" podID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerID="5bdd434b2ad930a6a62c12f7a0d1ace3ffabb443a40186c1723be61af43637c0" exitCode=0 Sep 10 13:37:50.099445 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.099034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" event={"ID":"fcd71ede-2461-411b-91e3-b7d06acf8650","Type":"ContainerDied","Data":"5bdd434b2ad930a6a62c12f7a0d1ace3ffabb443a40186c1723be61af43637c0"} Sep 10 13:37:50.101182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.101159 2559 generic.go:356] "Generic (PLEG): container finished" podID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerID="bc40e90c0e85190eec5f5a4be9696224ff4a336a103bae9958ff47d131268605" exitCode=0 Sep 10 13:37:50.101301 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.101275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" event={"ID":"f424e499-9a8e-4ac5-a4f6-eff65f0851f1","Type":"ContainerDied","Data":"bc40e90c0e85190eec5f5a4be9696224ff4a336a103bae9958ff47d131268605"} Sep 10 13:37:50.103492 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.103470 2559 generic.go:356] "Generic (PLEG): container finished" podID="43109b54-782b-49b6-92b8-038b5147f18d" containerID="3c2004e4afb5e3bff13c976fab8163813d06da3d485f0f2c5bff2eed40af7904" exitCode=0 Sep 10 13:37:50.103607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.103532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerDied","Data":"3c2004e4afb5e3bff13c976fab8163813d06da3d485f0f2c5bff2eed40af7904"} Sep 10 13:37:50.229821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.229794 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:50.359934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.359878 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util\") pod \"59416be3-865b-4ed6-b1c2-07e62868f881\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " Sep 10 13:37:50.360088 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.359937 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj6tx\" (UniqueName: \"kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx\") pod \"59416be3-865b-4ed6-b1c2-07e62868f881\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " Sep 10 13:37:50.360088 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.359956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle\") pod \"59416be3-865b-4ed6-b1c2-07e62868f881\" (UID: \"59416be3-865b-4ed6-b1c2-07e62868f881\") " Sep 10 13:37:50.360524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.360500 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle" (OuterVolumeSpecName: "bundle") pod "59416be3-865b-4ed6-b1c2-07e62868f881" (UID: "59416be3-865b-4ed6-b1c2-07e62868f881"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:50.362037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.362013 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx" (OuterVolumeSpecName: "kube-api-access-pj6tx") pod "59416be3-865b-4ed6-b1c2-07e62868f881" (UID: "59416be3-865b-4ed6-b1c2-07e62868f881"). InnerVolumeSpecName "kube-api-access-pj6tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:50.364704 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.364680 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util" (OuterVolumeSpecName: "util") pod "59416be3-865b-4ed6-b1c2-07e62868f881" (UID: "59416be3-865b-4ed6-b1c2-07e62868f881"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:50.461419 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.461396 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:50.461419 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.461417 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj6tx\" (UniqueName: \"kubernetes.io/projected/59416be3-865b-4ed6-b1c2-07e62868f881-kube-api-access-pj6tx\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:50.461538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:50.461426 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59416be3-865b-4ed6-b1c2-07e62868f881-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.110467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.110426 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" event={"ID":"59416be3-865b-4ed6-b1c2-07e62868f881","Type":"ContainerDied","Data":"9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654"} Sep 10 13:37:51.110876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.110487 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7142b64cd60da2240d7f40c09b20be648a0e39d89691acb4d09150267f3654" Sep 10 13:37:51.110876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.110446 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77rcfl" Sep 10 13:37:51.257879 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.257851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:51.307035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.307002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:51.310586 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.310565 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:51.369557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.369524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle\") pod \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " Sep 10 13:37:51.369723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.369585 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util\") pod \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " Sep 10 13:37:51.369723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.369633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4kdz\" (UniqueName: \"kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz\") pod \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\" (UID: \"f424e499-9a8e-4ac5-a4f6-eff65f0851f1\") " Sep 10 13:37:51.370051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.370023 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle" (OuterVolumeSpecName: "bundle") pod "f424e499-9a8e-4ac5-a4f6-eff65f0851f1" (UID: "f424e499-9a8e-4ac5-a4f6-eff65f0851f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.371686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.371660 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz" (OuterVolumeSpecName: "kube-api-access-z4kdz") pod "f424e499-9a8e-4ac5-a4f6-eff65f0851f1" (UID: "f424e499-9a8e-4ac5-a4f6-eff65f0851f1"). InnerVolumeSpecName "kube-api-access-z4kdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:51.375130 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.375104 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util" (OuterVolumeSpecName: "util") pod "f424e499-9a8e-4ac5-a4f6-eff65f0851f1" (UID: "f424e499-9a8e-4ac5-a4f6-eff65f0851f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.470551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470487 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle\") pod \"43109b54-782b-49b6-92b8-038b5147f18d\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " Sep 10 13:37:51.470551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470546 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrrcv\" (UniqueName: \"kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv\") pod \"43109b54-782b-49b6-92b8-038b5147f18d\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " Sep 10 13:37:51.470681 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470571 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util\") pod \"43109b54-782b-49b6-92b8-038b5147f18d\" (UID: \"43109b54-782b-49b6-92b8-038b5147f18d\") " Sep 10 13:37:51.470681 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470615 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle\") pod \"fcd71ede-2461-411b-91e3-b7d06acf8650\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " Sep 10 13:37:51.470793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470708 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util\") pod \"fcd71ede-2461-411b-91e3-b7d06acf8650\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " Sep 10 13:37:51.470793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.470756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqmm6\" (UniqueName: \"kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6\") pod \"fcd71ede-2461-411b-91e3-b7d06acf8650\" (UID: \"fcd71ede-2461-411b-91e3-b7d06acf8650\") " Sep 10 13:37:51.471706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.471599 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.471706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.471638 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.471706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.471653 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4kdz\" (UniqueName: \"kubernetes.io/projected/f424e499-9a8e-4ac5-a4f6-eff65f0851f1-kube-api-access-z4kdz\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.471890 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.471684 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle" (OuterVolumeSpecName: "bundle") pod "fcd71ede-2461-411b-91e3-b7d06acf8650" (UID: "fcd71ede-2461-411b-91e3-b7d06acf8650"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.472028 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.472003 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle" (OuterVolumeSpecName: "bundle") pod "43109b54-782b-49b6-92b8-038b5147f18d" (UID: "43109b54-782b-49b6-92b8-038b5147f18d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.473031 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.472996 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv" (OuterVolumeSpecName: "kube-api-access-qrrcv") pod "43109b54-782b-49b6-92b8-038b5147f18d" (UID: "43109b54-782b-49b6-92b8-038b5147f18d"). InnerVolumeSpecName "kube-api-access-qrrcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:51.479295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.473607 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6" (OuterVolumeSpecName: "kube-api-access-nqmm6") pod "fcd71ede-2461-411b-91e3-b7d06acf8650" (UID: "fcd71ede-2461-411b-91e3-b7d06acf8650"). InnerVolumeSpecName "kube-api-access-nqmm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:37:51.480556 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.480529 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util" (OuterVolumeSpecName: "util") pod "43109b54-782b-49b6-92b8-038b5147f18d" (UID: "43109b54-782b-49b6-92b8-038b5147f18d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.480840 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.480823 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util" (OuterVolumeSpecName: "util") pod "fcd71ede-2461-411b-91e3-b7d06acf8650" (UID: "fcd71ede-2461-411b-91e3-b7d06acf8650"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:37:51.572279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572257 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.572388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572295 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.572388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572304 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcd71ede-2461-411b-91e3-b7d06acf8650-util\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.572388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572312 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmm6\" (UniqueName: \"kubernetes.io/projected/fcd71ede-2461-411b-91e3-b7d06acf8650-kube-api-access-nqmm6\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.572388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572321 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43109b54-782b-49b6-92b8-038b5147f18d-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:51.572388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:51.572329 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrcv\" (UniqueName: \"kubernetes.io/projected/43109b54-782b-49b6-92b8-038b5147f18d-kube-api-access-qrrcv\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:37:52.117156 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.117129 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" Sep 10 13:37:52.117156 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.117143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf67hm" event={"ID":"43109b54-782b-49b6-92b8-038b5147f18d","Type":"ContainerDied","Data":"4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3"} Sep 10 13:37:52.117653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.117184 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c880a8edba3f2c72905fc1fb31a2b44e75ec3f0c007d8ab794166b921619cf3" Sep 10 13:37:52.119311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.119281 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" Sep 10 13:37:52.119440 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.119280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7tqc2" event={"ID":"fcd71ede-2461-411b-91e3-b7d06acf8650","Type":"ContainerDied","Data":"26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9"} Sep 10 13:37:52.119440 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.119433 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26575e5e29a27ab16df35dbc46a5bfa05ae4dddc85663b3aa16d211498a7fbb9" Sep 10 13:37:52.121425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.121396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" event={"ID":"f424e499-9a8e-4ac5-a4f6-eff65f0851f1","Type":"ContainerDied","Data":"01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d"} Sep 10 13:37:52.121425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.121422 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c655xn9" Sep 10 13:37:52.121569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.121441 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ccd4a5e6f3b4339be83740f086b64c2fb414460fca72a024a6a3ce8f0bc67d" Sep 10 13:37:52.900153 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.900118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:52.999158 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:52.999129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:55.114653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:55.114616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:55.905158 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:55.905126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:58.431084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431043 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6dh2p"] Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431418 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431432 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431448 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431454 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431465 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="extract" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431471 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="extract" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431479 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431485 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="pull" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431494 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="util" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431499 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="util" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431505 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="extract" Sep 10 13:37:58.431515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431511 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431534 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431540 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431550 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431556 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431567 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431573 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431580 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431585 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431590 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431596 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="util" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431605 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="pull" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431610 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="pull" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431665 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd71ede-2461-411b-91e3-b7d06acf8650" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431674 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43109b54-782b-49b6-92b8-038b5147f18d" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431681 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59416be3-865b-4ed6-b1c2-07e62868f881" containerName="extract" Sep 10 13:37:58.431886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.431686 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f424e499-9a8e-4ac5-a4f6-eff65f0851f1" containerName="extract" Sep 10 13:37:58.436318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.436297 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:37:58.439579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.439549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 13:37:58.439579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.439570 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 13:37:58.441090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.441066 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7bp4h\"" Sep 10 13:37:58.448621 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.448596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6dh2p"] Sep 10 13:37:58.519184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.519156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqp9z\" (UniqueName: \"kubernetes.io/projected/7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c-kube-api-access-xqp9z\") pod \"authorino-operator-76b478bd78-6dh2p\" (UID: \"7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c\") " pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:37:58.619610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.619579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqp9z\" (UniqueName: \"kubernetes.io/projected/7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c-kube-api-access-xqp9z\") pod \"authorino-operator-76b478bd78-6dh2p\" (UID: \"7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c\") " pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:37:58.633348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.633317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqp9z\" (UniqueName: \"kubernetes.io/projected/7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c-kube-api-access-xqp9z\") pod \"authorino-operator-76b478bd78-6dh2p\" (UID: \"7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c\") " pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:37:58.748601 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:58.748539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:37:59.027034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:59.027001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:37:59.040665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:59.040640 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-6dh2p"] Sep 10 13:37:59.042795 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:37:59.042767 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5fd2e3_f1ad_4bb1_8840_d226f33e1b5c.slice/crio-542cd67fc21cb3e7c49153b6008b8701fa3d676aa272a2725504640e9fb1a63f WatchSource:0}: Error finding container 542cd67fc21cb3e7c49153b6008b8701fa3d676aa272a2725504640e9fb1a63f: Status 404 returned error can't find the container with id 542cd67fc21cb3e7c49153b6008b8701fa3d676aa272a2725504640e9fb1a63f Sep 10 13:37:59.159141 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:59.159115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" event={"ID":"7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c","Type":"ContainerStarted","Data":"542cd67fc21cb3e7c49153b6008b8701fa3d676aa272a2725504640e9fb1a63f"} Sep 10 13:37:59.294484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:37:59.294415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:00.709016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.708979 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5"] Sep 10 13:38:00.716062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.716039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:00.720316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.720290 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7m4wl\"" Sep 10 13:38:00.728130 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.728102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5"] Sep 10 13:38:00.839075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.839039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7k2b\" (UniqueName: \"kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b\") pod \"limitador-operator-controller-manager-74495c69f8-mfqt5\" (UID: \"846b93cd-cb28-428b-8682-82a2feec434f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:00.939818 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.939782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7k2b\" (UniqueName: \"kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b\") pod \"limitador-operator-controller-manager-74495c69f8-mfqt5\" (UID: \"846b93cd-cb28-428b-8682-82a2feec434f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:00.961645 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.961580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7k2b\" (UniqueName: \"kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b\") pod \"limitador-operator-controller-manager-74495c69f8-mfqt5\" (UID: \"846b93cd-cb28-428b-8682-82a2feec434f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:00.965462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:00.965429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:01.029124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:01.029080 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:01.115076 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:01.115018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:01.791540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:01.791345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:01.791822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:01.791711 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5"] Sep 10 13:38:01.794427 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:38:01.794402 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846b93cd_cb28_428b_8682_82a2feec434f.slice/crio-99794accfa4f01fc295b4c399a5553377541356696a5730ac520b88720d88948 WatchSource:0}: Error finding container 99794accfa4f01fc295b4c399a5553377541356696a5730ac520b88720d88948: Status 404 returned error can't find the container with id 99794accfa4f01fc295b4c399a5553377541356696a5730ac520b88720d88948 Sep 10 13:38:02.031064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.031029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:02.124599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.124523 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:02.179839 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.179809 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" event={"ID":"846b93cd-cb28-428b-8682-82a2feec434f","Type":"ContainerStarted","Data":"99794accfa4f01fc295b4c399a5553377541356696a5730ac520b88720d88948"} Sep 10 13:38:02.181533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.181508 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" event={"ID":"7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c","Type":"ContainerStarted","Data":"a8dcd44197a2a28b482885f281d4ddc6c5d04e5a609af619b170f85f56f8f1a2"} Sep 10 13:38:02.181674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.181660 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:38:02.206080 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.206035 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" podStartSLOduration=1.65388129 podStartE2EDuration="4.206019811s" totalImagesPullingTime="2.552138521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:02.202496076 +0000 UTC m=+457.812280989" watchObservedRunningTime="2026-09-10 13:38:02.206019811 +0000 UTC m=+457.815804725" Sep 10 13:38:02.292361 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:02.292322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:04.194144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:04.194106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" event={"ID":"846b93cd-cb28-428b-8682-82a2feec434f","Type":"ContainerStarted","Data":"5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1"} Sep 10 13:38:04.194563 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:04.194257 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:04.216035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:04.215979 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" podStartSLOduration=2.838296441 podStartE2EDuration="4.215962888s" totalImagesPullingTime="1.377666447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:04.214556878 +0000 UTC m=+459.824341791" watchObservedRunningTime="2026-09-10 13:38:04.215962888 +0000 UTC m=+459.825747801" Sep 10 13:38:05.517018 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:05.516982 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:06.166345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:06.166302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:07.303318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.303279 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg"] Sep 10 13:38:07.307004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.306984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:07.309718 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.309692 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 13:38:07.309957 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.309942 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xxw9h\"" Sep 10 13:38:07.326865 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.326838 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg"] Sep 10 13:38:07.489222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.489192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zbq\" (UniqueName: \"kubernetes.io/projected/3541f750-d0f7-453c-92a9-f9ffe38a8741-kube-api-access-p8zbq\") pod \"dns-operator-controller-manager-7b84cd7c9c-4pspg\" (UID: \"3541f750-d0f7-453c-92a9-f9ffe38a8741\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:07.590201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.590133 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zbq\" (UniqueName: \"kubernetes.io/projected/3541f750-d0f7-453c-92a9-f9ffe38a8741-kube-api-access-p8zbq\") pod \"dns-operator-controller-manager-7b84cd7c9c-4pspg\" (UID: \"3541f750-d0f7-453c-92a9-f9ffe38a8741\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:07.599343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.599309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zbq\" (UniqueName: \"kubernetes.io/projected/3541f750-d0f7-453c-92a9-f9ffe38a8741-kube-api-access-p8zbq\") pod \"dns-operator-controller-manager-7b84cd7c9c-4pspg\" (UID: \"3541f750-d0f7-453c-92a9-f9ffe38a8741\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:07.618193 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.618169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:07.903888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:07.903865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg"] Sep 10 13:38:07.906345 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:38:07.906317 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3541f750_d0f7_453c_92a9_f9ffe38a8741.slice/crio-34e4cb356e6324be873a4ae3f2dc26470ebf5ea4c57ad0de7aa1bf9affd19824 WatchSource:0}: Error finding container 34e4cb356e6324be873a4ae3f2dc26470ebf5ea4c57ad0de7aa1bf9affd19824: Status 404 returned error can't find the container with id 34e4cb356e6324be873a4ae3f2dc26470ebf5ea4c57ad0de7aa1bf9affd19824 Sep 10 13:38:08.216582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:08.216515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" event={"ID":"3541f750-d0f7-453c-92a9-f9ffe38a8741","Type":"ContainerStarted","Data":"34e4cb356e6324be873a4ae3f2dc26470ebf5ea4c57ad0de7aa1bf9affd19824"} Sep 10 13:38:08.522729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:08.522693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:08.625955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:08.625906 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:09.140551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:09.140506 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:09.258496 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:09.258442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:10.229064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:10.229030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" event={"ID":"3541f750-d0f7-453c-92a9-f9ffe38a8741","Type":"ContainerStarted","Data":"51d7f4560ca9826acdcea00c81747850bde080e8d5e648ece96081c0564ad76c"} Sep 10 13:38:10.229493 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:10.229171 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:10.254589 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:10.254545 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" podStartSLOduration=1.6104885279999999 podStartE2EDuration="3.254532609s" totalImagesPullingTime="1.644044081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:10.252532932 +0000 UTC m=+465.862317844" watchObservedRunningTime="2026-09-10 13:38:10.254532609 +0000 UTC m=+465.864317523" Sep 10 13:38:12.124803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:12.124768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:12.257986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:12.257959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:12.799994 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:12.799957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:13.188997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:13.188921 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-6dh2p" Sep 10 13:38:13.364632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:13.364598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:15.202051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:15.202016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:16.000387 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:16.000349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:16.104295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:16.104261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:16.209422 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:16.209387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:21.237184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:21.237147 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-4pspg" Sep 10 13:38:22.509470 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:22.509440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:22.606552 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:22.606522 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:23.083989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.083955 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5"] Sep 10 13:38:23.088749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.088731 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.091810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.091787 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sztmq\"" Sep 10 13:38:23.163945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.163915 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5"] Sep 10 13:38:23.206232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.206205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.206392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.206305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4k6\" (UniqueName: \"kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.307174 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.307148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.307276 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.307202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4k6\" (UniqueName: \"kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.307538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.307518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.318139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.318114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4k6\" (UniqueName: \"kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6\") pod \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.398821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.398767 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:23.686822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.686797 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5"] Sep 10 13:38:23.689683 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:38:23.689651 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd99fca_a542_4535_a161_2aed088a0915.slice/crio-629ee46b8b6b73f7f4f99c06ed3f40006f3c5e5b5c580eacb1d2629cd4cfde50 WatchSource:0}: Error finding container 629ee46b8b6b73f7f4f99c06ed3f40006f3c5e5b5c580eacb1d2629cd4cfde50: Status 404 returned error can't find the container with id 629ee46b8b6b73f7f4f99c06ed3f40006f3c5e5b5c580eacb1d2629cd4cfde50 Sep 10 13:38:23.926795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.926770 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5"] Sep 10 13:38:23.935660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.934295 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5"] Sep 10 13:38:23.955702 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.955649 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:23.960331 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.960313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:23.976634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.976612 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:23.990994 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.990970 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5"] Sep 10 13:38:23.991209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.991166 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" podUID="846b93cd-cb28-428b-8682-82a2feec434f" containerName="manager" containerID="cri-o://5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1" gracePeriod=2 Sep 10 13:38:23.994403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:23.994356 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5"] Sep 10 13:38:24.015470 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.015442 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mfqt5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="846b93cd-cb28-428b-8682-82a2feec434f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.112632 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.112606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nx6\" (UniqueName: \"kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.112728 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.112714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.213832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.213759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nx6\" (UniqueName: \"kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.213977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.213843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.214192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.214173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.222187 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.222164 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.224457 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.224430 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mfqt5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="846b93cd-cb28-428b-8682-82a2feec434f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.228669 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.228647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nx6\" (UniqueName: \"kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6\") pod \"kuadrant-operator-controller-manager-5f685c496f-kvrgx\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.271637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.271620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:24.305146 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.305114 2559 generic.go:356] "Generic (PLEG): container finished" podID="846b93cd-cb28-428b-8682-82a2feec434f" containerID="5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1" exitCode=0 Sep 10 13:38:24.305248 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.305157 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.305248 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.305205 2559 scope.go:117] "RemoveContainer" containerID="5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1" Sep 10 13:38:24.307560 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.307531 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mfqt5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="846b93cd-cb28-428b-8682-82a2feec434f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.314807 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.314783 2559 scope.go:117] "RemoveContainer" containerID="5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1" Sep 10 13:38:24.315107 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.315090 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1\": container with ID starting with 5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1 not found: ID does not exist" containerID="5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1" Sep 10 13:38:24.315163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.315115 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1"} err="failed to get container status \"5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1\": rpc error: code = NotFound desc = could not find container \"5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1\": container with ID starting with 5da590186ccf0121188ad4356e1883e4a3de3a846bc17b02ef93b11358b8e6e1 not found: ID does not exist" Sep 10 13:38:24.415928 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.415895 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7k2b\" (UniqueName: \"kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b\") pod \"846b93cd-cb28-428b-8682-82a2feec434f\" (UID: \"846b93cd-cb28-428b-8682-82a2feec434f\") " Sep 10 13:38:24.418256 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.418213 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b" (OuterVolumeSpecName: "kube-api-access-x7k2b") pod "846b93cd-cb28-428b-8682-82a2feec434f" (UID: "846b93cd-cb28-428b-8682-82a2feec434f"). InnerVolumeSpecName "kube-api-access-x7k2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:38:24.516802 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.516776 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7k2b\" (UniqueName: \"kubernetes.io/projected/846b93cd-cb28-428b-8682-82a2feec434f-kube-api-access-x7k2b\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:38:24.561965 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.561939 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:24.562517 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:38:24.562486 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776acefb_ec42_42f9_bf84_6586cd438be6.slice/crio-568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d WatchSource:0}: Error finding container 568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d: Status 404 returned error can't find the container with id 568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d Sep 10 13:38:24.619788 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.619757 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mfqt5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="846b93cd-cb28-428b-8682-82a2feec434f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:24.808855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.808774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:24.906137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.906097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:24.920501 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:24.920472 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846b93cd-cb28-428b-8682-82a2feec434f" path="/var/lib/kubelet/pods/846b93cd-cb28-428b-8682-82a2feec434f/volumes" Sep 10 13:38:24.921092 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:24.921067 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mfqt5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="846b93cd-cb28-428b-8682-82a2feec434f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mfqt5" Sep 10 13:38:25.316120 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:25.316083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" event={"ID":"776acefb-ec42-42f9-bf84-6586cd438be6","Type":"ContainerStarted","Data":"568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d"} Sep 10 13:38:28.232246 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.232202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:28.336970 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.336934 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" podUID="6dd99fca-a542-4535-a161-2aed088a0915" containerName="manager" containerID="cri-o://bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2" gracePeriod=2 Sep 10 13:38:28.338939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.338910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:28.339407 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.339359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" event={"ID":"776acefb-ec42-42f9-bf84-6586cd438be6","Type":"ContainerStarted","Data":"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0"} Sep 10 13:38:28.339545 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.339514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:28.376317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.376276 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" podStartSLOduration=2.344187859 podStartE2EDuration="5.37626228s" totalImagesPullingTime="3.032074421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:28.373351411 +0000 UTC m=+483.983136325" watchObservedRunningTime="2026-09-10 13:38:28.37626228 +0000 UTC m=+483.986047194" Sep 10 13:38:28.582526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.582504 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:28.584875 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:28.584849 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="6dd99fca-a542-4535-a161-2aed088a0915" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:28.651988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.651963 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume\") pod \"6dd99fca-a542-4535-a161-2aed088a0915\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " Sep 10 13:38:28.652093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.652071 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td4k6\" (UniqueName: \"kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6\") pod \"6dd99fca-a542-4535-a161-2aed088a0915\" (UID: \"6dd99fca-a542-4535-a161-2aed088a0915\") " Sep 10 13:38:28.652251 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.652230 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6dd99fca-a542-4535-a161-2aed088a0915" (UID: "6dd99fca-a542-4535-a161-2aed088a0915"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:38:28.652323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.652307 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd99fca-a542-4535-a161-2aed088a0915-extensions-socket-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:38:28.654032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.654012 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6" (OuterVolumeSpecName: "kube-api-access-td4k6") pod "6dd99fca-a542-4535-a161-2aed088a0915" (UID: "6dd99fca-a542-4535-a161-2aed088a0915"). InnerVolumeSpecName "kube-api-access-td4k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:38:28.753576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.753556 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4k6\" (UniqueName: \"kubernetes.io/projected/6dd99fca-a542-4535-a161-2aed088a0915-kube-api-access-td4k6\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:38:28.860510 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.860452 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:28.918360 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.918335 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd99fca-a542-4535-a161-2aed088a0915" path="/var/lib/kubelet/pods/6dd99fca-a542-4535-a161-2aed088a0915/volumes" Sep 10 13:38:28.958641 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:28.958614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:29.346448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:29.346419 2559 generic.go:356] "Generic (PLEG): container finished" podID="6dd99fca-a542-4535-a161-2aed088a0915" containerID="bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2" exitCode=2 Sep 10 13:38:29.346875 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:29.346462 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:29.346875 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:29.346522 2559 scope.go:117] "RemoveContainer" containerID="bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2" Sep 10 13:38:29.351224 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:29.351197 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vtjx5\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="6dd99fca-a542-4535-a161-2aed088a0915" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vtjx5" Sep 10 13:38:29.355542 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:29.355524 2559 scope.go:117] "RemoveContainer" containerID="bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2" Sep 10 13:38:29.355800 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:29.355779 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2\": container with ID starting with bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2 not found: ID does not exist" containerID="bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2" Sep 10 13:38:29.355864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:29.355806 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2"} err="failed to get container status \"bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2\": rpc error: code = NotFound desc = could not find container \"bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2\": container with ID starting with bae55bc5a7f08b7987943027d29233e8764404bc1c6f31ec8ce0ed1524988fc2 not found: ID does not exist" Sep 10 13:38:32.781841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:32.781800 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:32.874402 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:32.874352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:33.235889 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:33.235812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:33.337117 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:33.337088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:34.981480 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:34.981440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:35.118145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:35.118111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:38.107965 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:38.107923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:38.236785 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:38.236752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:38.374615 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:38.374528 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:38.713996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:38.713898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:38.919795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:38.919766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:39.333270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:39.333239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:39.348947 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:39.348922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:39.526824 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:39.526794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:46.537732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:46.537699 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:46.700173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:46.700132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:51.762682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:51.762646 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:51.763070 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:51.762834 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" podUID="776acefb-ec42-42f9-bf84-6586cd438be6" containerName="manager" containerID="cri-o://b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0" gracePeriod=10 Sep 10 13:38:52.012511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.012489 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:52.134308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.134246 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume\") pod \"776acefb-ec42-42f9-bf84-6586cd438be6\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " Sep 10 13:38:52.134453 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.134400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6nx6\" (UniqueName: \"kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6\") pod \"776acefb-ec42-42f9-bf84-6586cd438be6\" (UID: \"776acefb-ec42-42f9-bf84-6586cd438be6\") " Sep 10 13:38:52.134740 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.134720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "776acefb-ec42-42f9-bf84-6586cd438be6" (UID: "776acefb-ec42-42f9-bf84-6586cd438be6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:38:52.136276 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.136256 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6" (OuterVolumeSpecName: "kube-api-access-v6nx6") pod "776acefb-ec42-42f9-bf84-6586cd438be6" (UID: "776acefb-ec42-42f9-bf84-6586cd438be6"). InnerVolumeSpecName "kube-api-access-v6nx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:38:52.235011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.234988 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6nx6\" (UniqueName: \"kubernetes.io/projected/776acefb-ec42-42f9-bf84-6586cd438be6-kube-api-access-v6nx6\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:38:52.235011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.235008 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776acefb-ec42-42f9-bf84-6586cd438be6-extensions-socket-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:38:52.472923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.472835 2559 generic.go:356] "Generic (PLEG): container finished" podID="776acefb-ec42-42f9-bf84-6586cd438be6" containerID="b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0" exitCode=0 Sep 10 13:38:52.472923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.472881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" event={"ID":"776acefb-ec42-42f9-bf84-6586cd438be6","Type":"ContainerDied","Data":"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0"} Sep 10 13:38:52.472923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.472889 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" Sep 10 13:38:52.473189 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.472939 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx" event={"ID":"776acefb-ec42-42f9-bf84-6586cd438be6","Type":"ContainerDied","Data":"568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d"} Sep 10 13:38:52.473189 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.472968 2559 scope.go:117] "RemoveContainer" containerID="b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0" Sep 10 13:38:52.482949 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.482933 2559 scope.go:117] "RemoveContainer" containerID="b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0" Sep 10 13:38:52.483225 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:52.483205 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0\": container with ID starting with b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0 not found: ID does not exist" containerID="b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0" Sep 10 13:38:52.483267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.483235 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0"} err="failed to get container status \"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0\": rpc error: code = NotFound desc = could not find container \"b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0\": container with ID starting with b2ab4f0a0ef889aa61bafe69501657de02c280f32f70347884435de2303538d0 not found: ID does not exist" Sep 10 13:38:52.499856 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:38:52.499829 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776acefb_ec42_42f9_bf84_6586cd438be6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776acefb_ec42_42f9_bf84_6586cd438be6.slice/crio-568dbea53f8da33b6b33acba180c8a7b607455c95356e831d1e6f0563040aa4d\": RecentStats: unable to find data in memory cache]" Sep 10 13:38:52.512362 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.512340 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:52.520508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.520486 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-kvrgx"] Sep 10 13:38:52.917181 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:52.917151 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776acefb-ec42-42f9-bf84-6586cd438be6" path="/var/lib/kubelet/pods/776acefb-ec42-42f9-bf84-6586cd438be6/volumes" Sep 10 13:38:56.282838 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.282808 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk"] Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283139 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776acefb-ec42-42f9-bf84-6586cd438be6" containerName="manager" Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283150 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="776acefb-ec42-42f9-bf84-6586cd438be6" containerName="manager" Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283168 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd99fca-a542-4535-a161-2aed088a0915" containerName="manager" Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283173 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd99fca-a542-4535-a161-2aed088a0915" containerName="manager" Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283190 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846b93cd-cb28-428b-8682-82a2feec434f" containerName="manager" Sep 10 13:38:56.283211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283195 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="846b93cd-cb28-428b-8682-82a2feec434f" containerName="manager" Sep 10 13:38:56.283427 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283243 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="776acefb-ec42-42f9-bf84-6586cd438be6" containerName="manager" Sep 10 13:38:56.283427 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283253 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd99fca-a542-4535-a161-2aed088a0915" containerName="manager" Sep 10 13:38:56.283427 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.283262 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="846b93cd-cb28-428b-8682-82a2feec434f" containerName="manager" Sep 10 13:38:56.287567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.287549 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.291050 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.291024 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-75qc2\"" Sep 10 13:38:56.300443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.300422 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk"] Sep 10 13:38:56.303059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.303036 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:56.367326 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8nj\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-kube-api-access-cx8nj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367645 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.367810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.367800 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.468850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8nj\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-kube-api-access-cx8nj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.468977 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.469057 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.469095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469507 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.469484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.469660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.469635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.471344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.471319 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.471657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.471635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.479696 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.479667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8nj\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-kube-api-access-cx8nj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.480512 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.480494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9f0176-0f4d-447a-a297-2c03237e1f9b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-fnzdk\" (UID: \"cf9f0176-0f4d-447a-a297-2c03237e1f9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.598821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.598764 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:56.888257 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.888231 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk"] Sep 10 13:38:56.889846 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:38:56.889810 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9f0176_0f4d_447a_a297_2c03237e1f9b.slice/crio-3ed33ee4f9455b5791725ca3ee508f00778a5d05cb2479123ec4fd06842b62e2 WatchSource:0}: Error finding container 3ed33ee4f9455b5791725ca3ee508f00778a5d05cb2479123ec4fd06842b62e2: Status 404 returned error can't find the container with id 3ed33ee4f9455b5791725ca3ee508f00778a5d05cb2479123ec4fd06842b62e2 Sep 10 13:38:56.892013 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.891979 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:38:56.892077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.892044 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:38:56.892117 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:56.892073 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:38:57.366038 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.366003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:38:57.509390 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.509330 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" event={"ID":"cf9f0176-0f4d-447a-a297-2c03237e1f9b","Type":"ContainerStarted","Data":"8bf98165d8d6d560c9c6deca37fb337537f9874e304bde600508e3dd2cf12acd"} Sep 10 13:38:57.509545 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.509401 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" event={"ID":"cf9f0176-0f4d-447a-a297-2c03237e1f9b","Type":"ContainerStarted","Data":"3ed33ee4f9455b5791725ca3ee508f00778a5d05cb2479123ec4fd06842b62e2"} Sep 10 13:38:57.551213 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.551161 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podStartSLOduration=1.5511468800000001 podStartE2EDuration="1.55114688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:57.550955045 +0000 UTC m=+513.160739957" watchObservedRunningTime="2026-09-10 13:38:57.55114688 +0000 UTC m=+513.160931793" Sep 10 13:38:57.599232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.599200 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:57.603935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:57.603910 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:58.515379 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:58.515347 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:38:58.516215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:38:58.516195 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:39:07.269398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.269341 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:07.273557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.273527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.273557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.273549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:07.276553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.276532 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 13:39:07.276672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.276567 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ph8jr\"" Sep 10 13:39:07.283622 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.283601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:07.354349 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.354306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.354528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.354376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpgt\" (UniqueName: \"kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.374168 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.374138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:07.375285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.375256 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:07.455277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.455247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.455441 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.455287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpgt\" (UniqueName: \"kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.455872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.455847 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.467819 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.467760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:07.469009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.468845 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpgt\" (UniqueName: \"kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt\") pod \"limitador-limitador-8588749977-7gwbn\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.584672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.584579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:07.679783 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.679751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:39:07.684010 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.683993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.694507 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.694487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:39:07.736869 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.736843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:39:07.759207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.759182 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mcn\" (UniqueName: \"kubernetes.io/projected/c494eafb-1357-4883-a003-50b78205641f-kube-api-access-s4mcn\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.759317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.759220 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c494eafb-1357-4883-a003-50b78205641f-config-file\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.860226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.860174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mcn\" (UniqueName: \"kubernetes.io/projected/c494eafb-1357-4883-a003-50b78205641f-kube-api-access-s4mcn\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.860226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.860211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c494eafb-1357-4883-a003-50b78205641f-config-file\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.860782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.860764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c494eafb-1357-4883-a003-50b78205641f-config-file\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.866233 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.866222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:07.868435 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:39:07.868410 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1639e0f9_aa21_4273_acf5_0e5a36389dcc.slice/crio-f55eb55b87e47d52ac781929ed8f8f656d7561792ca86dd673675efca353ba24 WatchSource:0}: Error finding container f55eb55b87e47d52ac781929ed8f8f656d7561792ca86dd673675efca353ba24: Status 404 returned error can't find the container with id f55eb55b87e47d52ac781929ed8f8f656d7561792ca86dd673675efca353ba24 Sep 10 13:39:07.878716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.878696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mcn\" (UniqueName: \"kubernetes.io/projected/c494eafb-1357-4883-a003-50b78205641f-kube-api-access-s4mcn\") pod \"limitador-limitador-84994f7ffc-zct4z\" (UID: \"c494eafb-1357-4883-a003-50b78205641f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:07.994211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:07.994188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:08.226694 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:08.226611 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:08.266009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:08.265987 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:39:08.267922 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:39:08.267898 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc494eafb_1357_4883_a003_50b78205641f.slice/crio-31e7fc80fa6b6bb132b124b361979f9a34e91c918e52732938c42794e5d0abb6 WatchSource:0}: Error finding container 31e7fc80fa6b6bb132b124b361979f9a34e91c918e52732938c42794e5d0abb6: Status 404 returned error can't find the container with id 31e7fc80fa6b6bb132b124b361979f9a34e91c918e52732938c42794e5d0abb6 Sep 10 13:39:08.327003 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:08.326981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:08.571935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:08.571904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" event={"ID":"c494eafb-1357-4883-a003-50b78205641f","Type":"ContainerStarted","Data":"31e7fc80fa6b6bb132b124b361979f9a34e91c918e52732938c42794e5d0abb6"} Sep 10 13:39:08.573196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:08.573173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" event={"ID":"1639e0f9-aa21-4273-acf5-0e5a36389dcc","Type":"ContainerStarted","Data":"f55eb55b87e47d52ac781929ed8f8f656d7561792ca86dd673675efca353ba24"} Sep 10 13:39:10.587582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:10.587541 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" event={"ID":"c494eafb-1357-4883-a003-50b78205641f","Type":"ContainerStarted","Data":"a46c7b8ed21ad91b79e828c27b53af3c5d938c1ebcef91fb746421f804fa4a94"} Sep 10 13:39:10.587963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:10.587615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:10.625675 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:10.625621 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" podStartSLOduration=2.125998721 podStartE2EDuration="3.625605s" totalImagesPullingTime="1.499606279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:10.623495154 +0000 UTC m=+526.233280068" watchObservedRunningTime="2026-09-10 13:39:10.625605 +0000 UTC m=+526.235389938" Sep 10 13:39:11.594962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:11.594922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" event={"ID":"1639e0f9-aa21-4273-acf5-0e5a36389dcc","Type":"ContainerStarted","Data":"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686"} Sep 10 13:39:11.595485 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:11.594996 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:11.618309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:11.618261 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" podStartSLOduration=1.793612432 podStartE2EDuration="4.618247676s" totalImagesPullingTime="2.824635244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:11.615021658 +0000 UTC m=+527.224806572" watchObservedRunningTime="2026-09-10 13:39:11.618247676 +0000 UTC m=+527.228032588" Sep 10 13:39:21.595787 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:21.595710 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-zct4z" Sep 10 13:39:21.669535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:21.669507 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:21.669731 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:21.669703 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" podUID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" containerName="limitador" containerID="cri-o://06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686" gracePeriod=30 Sep 10 13:39:21.670320 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:21.670268 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:22.214629 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.214604 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:22.370608 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.370584 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file\") pod \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " Sep 10 13:39:22.370756 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.370617 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgpgt\" (UniqueName: \"kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt\") pod \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\" (UID: \"1639e0f9-aa21-4273-acf5-0e5a36389dcc\") " Sep 10 13:39:22.371011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.370985 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file" (OuterVolumeSpecName: "config-file") pod "1639e0f9-aa21-4273-acf5-0e5a36389dcc" (UID: "1639e0f9-aa21-4273-acf5-0e5a36389dcc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:39:22.372681 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.372663 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt" (OuterVolumeSpecName: "kube-api-access-zgpgt") pod "1639e0f9-aa21-4273-acf5-0e5a36389dcc" (UID: "1639e0f9-aa21-4273-acf5-0e5a36389dcc"). InnerVolumeSpecName "kube-api-access-zgpgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:39:22.471951 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.471924 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1639e0f9-aa21-4273-acf5-0e5a36389dcc-config-file\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:39:22.471951 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.471948 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpgt\" (UniqueName: \"kubernetes.io/projected/1639e0f9-aa21-4273-acf5-0e5a36389dcc-kube-api-access-zgpgt\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:39:22.661302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.661229 2559 generic.go:356] "Generic (PLEG): container finished" podID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" containerID="06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686" exitCode=0 Sep 10 13:39:22.661302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.661276 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" Sep 10 13:39:22.661766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.661316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" event={"ID":"1639e0f9-aa21-4273-acf5-0e5a36389dcc","Type":"ContainerDied","Data":"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686"} Sep 10 13:39:22.661766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.661389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7gwbn" event={"ID":"1639e0f9-aa21-4273-acf5-0e5a36389dcc","Type":"ContainerDied","Data":"f55eb55b87e47d52ac781929ed8f8f656d7561792ca86dd673675efca353ba24"} Sep 10 13:39:22.661766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.661413 2559 scope.go:117] "RemoveContainer" containerID="06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686" Sep 10 13:39:22.671141 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.670975 2559 scope.go:117] "RemoveContainer" containerID="06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686" Sep 10 13:39:22.671237 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:39:22.671218 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686\": container with ID starting with 06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686 not found: ID does not exist" containerID="06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686" Sep 10 13:39:22.671281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.671244 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686"} err="failed to get container status \"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686\": rpc error: code = NotFound desc = could not find container \"06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686\": container with ID starting with 06fcf7bf6fc85a27530e564df29c7a9494975376bff7579218c3c69249099686 not found: ID does not exist" Sep 10 13:39:22.690353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.690327 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:22.694561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.694541 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7gwbn"] Sep 10 13:39:22.918720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:22.918657 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" path="/var/lib/kubelet/pods/1639e0f9-aa21-4273-acf5-0e5a36389dcc/volumes" Sep 10 13:39:25.120544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.120451 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:25.214469 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.214434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:25.610203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.610163 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9qmff"] Sep 10 13:39:25.610672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.610652 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" containerName="limitador" Sep 10 13:39:25.610672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.610671 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" containerName="limitador" Sep 10 13:39:25.610810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.610753 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1639e0f9-aa21-4273-acf5-0e5a36389dcc" containerName="limitador" Sep 10 13:39:25.615420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.615395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.617918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.617892 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 13:39:25.617918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.617907 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 13:39:25.618705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.618682 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kjgv7\"" Sep 10 13:39:25.618705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.618701 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 13:39:25.624741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.624715 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9qmff"] Sep 10 13:39:25.643092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.643047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:25.696449 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.696428 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/975629be-325e-438b-9ce7-8ff3115c4497-data\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.696538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.696491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6prf\" (UniqueName: \"kubernetes.io/projected/975629be-325e-438b-9ce7-8ff3115c4497-kube-api-access-f6prf\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.797262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.797234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/975629be-325e-438b-9ce7-8ff3115c4497-data\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.797387 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.797305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6prf\" (UniqueName: \"kubernetes.io/projected/975629be-325e-438b-9ce7-8ff3115c4497-kube-api-access-f6prf\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.797597 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.797580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/975629be-325e-438b-9ce7-8ff3115c4497-data\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.807543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.807520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6prf\" (UniqueName: \"kubernetes.io/projected/975629be-325e-438b-9ce7-8ff3115c4497-kube-api-access-f6prf\") pod \"postgres-69fb4594cf-9qmff\" (UID: \"975629be-325e-438b-9ce7-8ff3115c4497\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:25.928695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:25.928643 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:26.210203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:26.210180 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9qmff"] Sep 10 13:39:26.211705 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:39:26.211673 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975629be_325e_438b_9ce7_8ff3115c4497.slice/crio-5bafeedf68230f863fa2f65d871277c783c279ab4fe8b6b6e98770fe4aac626c WatchSource:0}: Error finding container 5bafeedf68230f863fa2f65d871277c783c279ab4fe8b6b6e98770fe4aac626c: Status 404 returned error can't find the container with id 5bafeedf68230f863fa2f65d871277c783c279ab4fe8b6b6e98770fe4aac626c Sep 10 13:39:26.687086 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:26.687057 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" event={"ID":"975629be-325e-438b-9ce7-8ff3115c4497","Type":"ContainerStarted","Data":"5bafeedf68230f863fa2f65d871277c783c279ab4fe8b6b6e98770fe4aac626c"} Sep 10 13:39:31.721646 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:31.721605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" event={"ID":"975629be-325e-438b-9ce7-8ff3115c4497","Type":"ContainerStarted","Data":"e888939be5104bb001d84a86b525fbc86a97cfd4c4b22162281520518b5069e9"} Sep 10 13:39:31.722053 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:31.721730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:31.751770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:31.751712 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" podStartSLOduration=1.9263462150000001 podStartE2EDuration="6.751692545s" totalImagesPullingTime="4.82534633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:31.750620517 +0000 UTC m=+547.360405430" watchObservedRunningTime="2026-09-10 13:39:31.751692545 +0000 UTC m=+547.361477459" Sep 10 13:39:32.248731 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:32.248693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:37.757499 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:37.757470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9qmff" Sep 10 13:39:38.159970 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:38.159874 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:38.259988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:38.259952 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:45.864483 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:45.864444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:45.980668 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:45.980637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:46.069828 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:46.069781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:46.323962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:46.323932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:49.465832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:49.465787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:39:49.553630 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:39:49.553600 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:21.158325 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:21.158287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:22.595979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:22.595903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:22.694090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:22.694051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:22.770313 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:22.770280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:23.249270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.249225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:23.345980 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.345947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:23.397398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.396631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:40:23.418685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.418657 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7wzmf"] Sep 10 13:40:23.422122 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.422097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.425730 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.425710 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 13:40:23.425921 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.425902 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5bdb5\"" Sep 10 13:40:23.439257 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.439235 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7wzmf"] Sep 10 13:40:23.494012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.493979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6d5z\" (UniqueName: \"kubernetes.io/projected/844b6c0b-9f1b-4270-b728-adad14093bec-kube-api-access-n6d5z\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.494123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.494103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b6c0b-9f1b-4270-b728-adad14093bec-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.594915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.594882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6d5z\" (UniqueName: \"kubernetes.io/projected/844b6c0b-9f1b-4270-b728-adad14093bec-kube-api-access-n6d5z\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.595085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.594933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b6c0b-9f1b-4270-b728-adad14093bec-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.595607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.595586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b6c0b-9f1b-4270-b728-adad14093bec-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.607380 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.607340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6d5z\" (UniqueName: \"kubernetes.io/projected/844b6c0b-9f1b-4270-b728-adad14093bec-kube-api-access-n6d5z\") pod \"payload-processing-5b54b7fbb7-7wzmf\" (UID: \"844b6c0b-9f1b-4270-b728-adad14093bec\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.732338 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.732307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:23.749637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:23.749601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:24.011865 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:24.011832 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7wzmf"] Sep 10 13:40:24.013081 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:40:24.013051 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844b6c0b_9f1b_4270_b728_adad14093bec.slice/crio-7111d2df4f86a3b5b76f0ca144b4a7cb5135b566355290c2d199bfe964ab4392 WatchSource:0}: Error finding container 7111d2df4f86a3b5b76f0ca144b4a7cb5135b566355290c2d199bfe964ab4392: Status 404 returned error can't find the container with id 7111d2df4f86a3b5b76f0ca144b4a7cb5135b566355290c2d199bfe964ab4392 Sep 10 13:40:24.873922 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:24.873890 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:40:24.877514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:24.877486 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:40:25.025032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:25.024993 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" event={"ID":"844b6c0b-9f1b-4270-b728-adad14093bec","Type":"ContainerStarted","Data":"7111d2df4f86a3b5b76f0ca144b4a7cb5135b566355290c2d199bfe964ab4392"} Sep 10 13:40:27.061935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:27.061896 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" event={"ID":"844b6c0b-9f1b-4270-b728-adad14093bec","Type":"ContainerStarted","Data":"3228e489f209281b803ee959da6cf154a488f2d6a7dfcc66b3b83e802b3f29b2"} Sep 10 13:40:27.062481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:27.062006 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:27.083795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:27.083745 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" podStartSLOduration=1.788621126 podStartE2EDuration="4.083729721s" totalImagesPullingTime="2.295108595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:27.081016706 +0000 UTC m=+602.690801616" watchObservedRunningTime="2026-09-10 13:40:27.083729721 +0000 UTC m=+602.693514636" Sep 10 13:40:33.097340 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.097307 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerID="8bf98165d8d6d560c9c6deca37fb337537f9874e304bde600508e3dd2cf12acd" exitCode=0 Sep 10 13:40:33.097818 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.097388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" event={"ID":"cf9f0176-0f4d-447a-a297-2c03237e1f9b","Type":"ContainerDied","Data":"8bf98165d8d6d560c9c6deca37fb337537f9874e304bde600508e3dd2cf12acd"} Sep 10 13:40:33.097818 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.097742 2559 scope.go:117] "RemoveContainer" containerID="8bf98165d8d6d560c9c6deca37fb337537f9874e304bde600508e3dd2cf12acd" Sep 10 13:40:33.098753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.098725 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:40:33.098821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.098803 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:40:33.098871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:33.098856 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:40:34.104864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.104830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" event={"ID":"cf9f0176-0f4d-447a-a297-2c03237e1f9b","Type":"ContainerStarted","Data":"f0834d6a509c2e823824c5034d34e2c55302e95e7e5a4edc76d28c3e3de32423"} Sep 10 13:40:34.211137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.211097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:34.305313 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.305275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:34.599562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.599526 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:40:34.599755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.599729 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fnzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:40:34.599836 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.599788 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podUID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 13:40:34.970052 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:34.969965 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:35.070579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:35.070534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:35.599851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:35.599813 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fnzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:40:35.600221 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:35.599872 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podUID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 13:40:36.599526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:36.599492 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fnzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:40:36.599731 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:36.599552 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podUID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 13:40:37.599833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:37.599798 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fnzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:40:37.600209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:37.599887 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podUID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 13:40:38.070267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:38.070240 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7wzmf" Sep 10 13:40:38.599337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:38.599295 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-fnzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 10 13:40:38.599549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:38.599359 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" podUID="cf9f0176-0f4d-447a-a297-2c03237e1f9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 10 13:40:39.603220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:39.603188 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:40:39.603716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:39.603606 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:40:39.604216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:39.604193 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-fnzdk" Sep 10 13:40:42.303342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.303297 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm"] Sep 10 13:40:42.310253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.310222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.312703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.312678 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 13:40:42.314055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.313868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 13:40:42.314055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.313891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zm5zz\"" Sep 10 13:40:42.314585 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.314563 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 13:40:42.318228 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.318191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm"] Sep 10 13:40:42.337738 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.337849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.337849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.337934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.337934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f706a784-b20a-4cc0-af04-2c8f39b61a76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.338003 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.337932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggxj\" (UniqueName: \"kubernetes.io/projected/f706a784-b20a-4cc0-af04-2c8f39b61a76-kube-api-access-hggxj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.340171 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.340146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:42.439183 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f706a784-b20a-4cc0-af04-2c8f39b61a76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439612 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggxj\" (UniqueName: \"kubernetes.io/projected/f706a784-b20a-4cc0-af04-2c8f39b61a76-kube-api-access-hggxj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439725 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.439832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.439814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.441983 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.441957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f706a784-b20a-4cc0-af04-2c8f39b61a76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.442820 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.442787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f706a784-b20a-4cc0-af04-2c8f39b61a76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.448511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.448486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggxj\" (UniqueName: \"kubernetes.io/projected/f706a784-b20a-4cc0-af04-2c8f39b61a76-kube-api-access-hggxj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm\" (UID: \"f706a784-b20a-4cc0-af04-2c8f39b61a76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.622622 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.622525 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:42.801909 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.801874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:40:42.903031 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.902938 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm"] Sep 10 13:40:42.905745 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:40:42.905720 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf706a784_b20a_4cc0_af04_2c8f39b61a76.slice/crio-0c60631bfd74a2a85e75f4f33b13ebf133b937b0a8fa782ed8ee61c1816832f9 WatchSource:0}: Error finding container 0c60631bfd74a2a85e75f4f33b13ebf133b937b0a8fa782ed8ee61c1816832f9: Status 404 returned error can't find the container with id 0c60631bfd74a2a85e75f4f33b13ebf133b937b0a8fa782ed8ee61c1816832f9 Sep 10 13:40:42.907145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:42.907124 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:40:43.156776 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:43.156689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" event={"ID":"f706a784-b20a-4cc0-af04-2c8f39b61a76","Type":"ContainerStarted","Data":"0c60631bfd74a2a85e75f4f33b13ebf133b937b0a8fa782ed8ee61c1816832f9"} Sep 10 13:40:44.164130 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:44.164096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" event={"ID":"f706a784-b20a-4cc0-af04-2c8f39b61a76","Type":"ContainerStarted","Data":"d58fdaf7f4292eda75214ac3ac0922b8e08af48e424a45bf89b3379b6ca31d36"} Sep 10 13:40:44.193657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:44.193608 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" podStartSLOduration=1.075550487 podStartE2EDuration="2.193594456s" totalImagesPullingTime="1.118043969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:44.193543006 +0000 UTC m=+619.803327913" watchObservedRunningTime="2026-09-10 13:40:44.193594456 +0000 UTC m=+619.803379368" Sep 10 13:40:52.083381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:52.083286 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:40:52.622827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:52.622797 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:52.622990 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:52.622851 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:52.635357 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:52.635333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:40:52.795697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:52.795664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:40:53.231665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:40:53.231632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm" Sep 10 13:41:16.145272 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:16.145236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:17.136540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:17.136504 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:41:35.976322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:35.976276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:37.214889 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:37.214851 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:41:42.396847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:42.396804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:43.989708 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:43.989672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:41:56.505551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:56.505516 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:41:58.307766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:41:58.307731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:42:09.170746 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:42:09.170707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:42:12.056986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:42:12.056956 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:43:11.081127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:11.081088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:11.182753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:11.182717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:12.347100 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:12.347066 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:12.454883 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:12.454851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:36.054392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.054343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:36.159032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.158996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:36.389596 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.389512 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:43:36.393138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.393110 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.395866 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.395839 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f1d168-openshift-default-dockercfg-5tv9x\"" Sep 10 13:43:36.412479 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.412456 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:43:36.449412 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.449382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:36.522761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.522938 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.522938 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vr2\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.522938 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.523083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.523083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.522995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.523083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.523037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.523198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.523093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.523198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.523113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.623925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.623899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.623932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.623971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.623988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624047 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vr2\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624171 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624491 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.624834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.624820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.626524 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.626504 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.626732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.626714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.635672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.635648 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.635672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.635665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vr2\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2\") pod \"e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:36.703542 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:36.703477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:37.028134 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.028110 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:43:37.030208 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:37.030178 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7254af13_d1f5_4022_bfb2_71dd0086a4ae.slice/crio-6587e5ba6c7dc6925b2c63a97409508371c604001bd42597a55bae00053fcc61 WatchSource:0}: Error finding container 6587e5ba6c7dc6925b2c63a97409508371c604001bd42597a55bae00053fcc61: Status 404 returned error can't find the container with id 6587e5ba6c7dc6925b2c63a97409508371c604001bd42597a55bae00053fcc61 Sep 10 13:43:37.032198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.032163 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:37.032254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.032228 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:37.032290 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.032257 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:37.162555 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.162514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerStarted","Data":"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e"} Sep 10 13:43:37.162897 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.162586 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerStarted","Data":"6587e5ba6c7dc6925b2c63a97409508371c604001bd42597a55bae00053fcc61"} Sep 10 13:43:37.189101 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.189047 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podStartSLOduration=1.189031218 podStartE2EDuration="1.189031218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:37.1882074 +0000 UTC m=+792.797992315" watchObservedRunningTime="2026-09-10 13:43:37.189031218 +0000 UTC m=+792.798816130" Sep 10 13:43:37.704091 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.704056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:37.768543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:37.768515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:38.708734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:38.708706 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:38.782102 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:38.782075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:38.896179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:38.896139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:39.164921 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:39.164894 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:39.174787 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:39.174761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:39.175696 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:39.175676 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:39.265932 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:39.265897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:40.192873 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:40.192837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:43:40.395648 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:40.395615 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:40.497191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:40.497101 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:40.890168 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:40.890128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:40.985571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:40.985526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:42.379521 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.379489 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-743aac"] Sep 10 13:43:42.383620 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.383603 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:42.386034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.386016 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-g8rj6\"" Sep 10 13:43:42.390491 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.390470 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-743aac"] Sep 10 13:43:42.480035 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.480003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74sjx\" (UniqueName: \"kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx\") pod \"test-curl-7853-743aac\" (UID: \"5a404b91-46fa-47db-825c-28f1fd5e2ed5\") " pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:42.580876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.580847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74sjx\" (UniqueName: \"kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx\") pod \"test-curl-7853-743aac\" (UID: \"5a404b91-46fa-47db-825c-28f1fd5e2ed5\") " pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:42.597643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.597620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74sjx\" (UniqueName: \"kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx\") pod \"test-curl-7853-743aac\" (UID: \"5a404b91-46fa-47db-825c-28f1fd5e2ed5\") " pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:42.689319 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.689255 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:42.694102 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.694085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:42.973070 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:42.973044 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-743aac"] Sep 10 13:43:42.975088 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:42.975054 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a404b91_46fa_47db_825c_28f1fd5e2ed5.slice/crio-cc2758afececc1646fd800b959619f9d208990b28cb2be53686eee4c601d9303 WatchSource:0}: Error finding container cc2758afececc1646fd800b959619f9d208990b28cb2be53686eee4c601d9303: Status 404 returned error can't find the container with id cc2758afececc1646fd800b959619f9d208990b28cb2be53686eee4c601d9303 Sep 10 13:43:43.201148 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:43.201114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-743aac" event={"ID":"5a404b91-46fa-47db-825c-28f1fd5e2ed5","Type":"ContainerStarted","Data":"cc2758afececc1646fd800b959619f9d208990b28cb2be53686eee4c601d9303"} Sep 10 13:43:45.053123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.053093 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:45.173785 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.173748 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:45.213147 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.213115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-743aac" event={"ID":"5a404b91-46fa-47db-825c-28f1fd5e2ed5","Type":"ContainerStarted","Data":"6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9"} Sep 10 13:43:45.248327 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.248278 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7853-743aac" podStartSLOduration=1.935254015 podStartE2EDuration="3.248262754s" totalImagesPullingTime="1.313008739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:45.247537356 +0000 UTC m=+800.857322269" watchObservedRunningTime="2026-09-10 13:43:45.248262754 +0000 UTC m=+800.858047667" Sep 10 13:43:45.344809 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.344717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:45.669561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.669483 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:43:45.673859 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.673836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.677982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.677960 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e9a01385-openshift-default-dockercfg-k5wmd\"" Sep 10 13:43:45.690445 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.690419 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:43:45.763628 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.763545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:45.812808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.812773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.812996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.812819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.812996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.812869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.812996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.812897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.812996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.812930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.813179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.813012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.813179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.813053 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz479\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.813179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.813081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.813179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.813140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz479\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.913997 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.913954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.914392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.914345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.914478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.914428 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.914478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.914447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.914625 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.914588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.914733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.914622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.916711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.916688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.916991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.916968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.923349 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.923298 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.923770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.923746 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz479\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479\") pod \"e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:45.985652 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.985626 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-743aac"] Sep 10 13:43:45.987861 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.987824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:45.990448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.990420 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-743aac"] Sep 10 13:43:45.998665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:45.998643 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:46.219215 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:43:46.219137 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7853-743aac" container="test-curl-7853-743aac" Sep 10 13:43:46.288913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:46.288883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:43:46.290313 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:46.290289 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda873df74_dfcd_4e31_afd6_06b372fd9d79.slice/crio-8b4a59586854f6e16c04a92c71a3c26578c496f06ccc4a4b086298d4cf579c27 WatchSource:0}: Error finding container 8b4a59586854f6e16c04a92c71a3c26578c496f06ccc4a4b086298d4cf579c27: Status 404 returned error can't find the container with id 8b4a59586854f6e16c04a92c71a3c26578c496f06ccc4a4b086298d4cf579c27 Sep 10 13:43:46.293161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:46.293121 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:46.293250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:46.293190 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:46.293250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:46.293220 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:46.533384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:46.533341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:47.226264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.226178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerStarted","Data":"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05"} Sep 10 13:43:47.226264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.226243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerStarted","Data":"8b4a59586854f6e16c04a92c71a3c26578c496f06ccc4a4b086298d4cf579c27"} Sep 10 13:43:47.226838 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.226359 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7853-743aac" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" containerName="test-curl-7853-743aac" containerID="cri-o://6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9" gracePeriod=2 Sep 10 13:43:47.266675 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.266628 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podStartSLOduration=2.266611477 podStartE2EDuration="2.266611477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:47.263585532 +0000 UTC m=+802.873370445" watchObservedRunningTime="2026-09-10 13:43:47.266611477 +0000 UTC m=+802.876396389" Sep 10 13:43:47.943958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.943921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:47.999541 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:47.999512 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:48.004442 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.004420 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:48.040099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.040076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:48.232264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.232181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:48.233202 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.233181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:43:48.761467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.761434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:48.888864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:48.888823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:49.369426 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:49.369292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:49.473906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:49.473876 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:49.553423 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:49.553385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74sjx\" (UniqueName: \"kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx\") pod \"5a404b91-46fa-47db-825c-28f1fd5e2ed5\" (UID: \"5a404b91-46fa-47db-825c-28f1fd5e2ed5\") " Sep 10 13:43:49.555627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:49.555599 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx" (OuterVolumeSpecName: "kube-api-access-74sjx") pod "5a404b91-46fa-47db-825c-28f1fd5e2ed5" (UID: "5a404b91-46fa-47db-825c-28f1fd5e2ed5"). InnerVolumeSpecName "kube-api-access-74sjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:43:49.654493 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:49.654454 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74sjx\" (UniqueName: \"kubernetes.io/projected/5a404b91-46fa-47db-825c-28f1fd5e2ed5-kube-api-access-74sjx\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:43:50.254533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.254481 2559 generic.go:356] "Generic (PLEG): container finished" podID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" containerID="6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9" exitCode=137 Sep 10 13:43:50.254533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.254525 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:50.254815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.254574 2559 scope.go:117] "RemoveContainer" containerID="6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9" Sep 10 13:43:50.264336 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.264316 2559 scope.go:117] "RemoveContainer" containerID="6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9" Sep 10 13:43:50.264633 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:43:50.264602 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9\": container with ID starting with 6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9 not found: ID does not exist" containerID="6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9" Sep 10 13:43:50.264692 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.264643 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9"} err="failed to get container status \"6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9\": rpc error: code = NotFound desc = could not find container \"6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9\": container with ID starting with 6c684bf8ab310d6ad2046cbf90519e47ac348db11c09a31ba6f660db85df68b9 not found: ID does not exist" Sep 10 13:43:50.264987 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:43:50.264969 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-743aac\" is forbidden: User \"system:node:ip-10-0-133-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-208.ec2.internal' and this object" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" pod="opendatahub/test-curl-7853-743aac" Sep 10 13:43:50.436394 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.436084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:50.535093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.535052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:50.681400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.681334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:50.924796 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:50.924713 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" path="/var/lib/kubelet/pods/5a404b91-46fa-47db-825c-28f1fd5e2ed5/volumes" Sep 10 13:43:51.147392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:51.146095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:51.245446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:51.243339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:51.640110 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:51.640072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:52.836049 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:52.836016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:52.939920 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:52.939885 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:53.297048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.297014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:53.479760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.479729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:53.574962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.574871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:53.673028 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.672994 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:43:53.673409 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.673396 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" containerName="test-curl-7853-743aac" Sep 10 13:43:53.673460 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.673411 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" containerName="test-curl-7853-743aac" Sep 10 13:43:53.673496 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.673481 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a404b91-46fa-47db-825c-28f1fd5e2ed5" containerName="test-curl-7853-743aac" Sep 10 13:43:53.677813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.677796 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.680561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.680540 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ba18dc-dockercfg-thrdj\"" Sep 10 13:43:53.680682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.680540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ba18dc\"" Sep 10 13:43:53.687974 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.687952 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:43:53.762723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.762680 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:43:53.766418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.766395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:53.777795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.777761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:43:53.790279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.790248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.790433 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.790357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpdl\" (UniqueName: \"kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.891695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.891596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpdl\" (UniqueName: \"kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.891695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.891651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghmg\" (UniqueName: \"kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:53.891695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.891696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:53.892276 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.891716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.892427 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.892405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.901957 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.901928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpdl\" (UniqueName: \"kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl\") pod \"payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.957392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.957349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:53.988737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.988710 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:53.993012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.992989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghmg\" (UniqueName: \"kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:53.993088 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.993055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:53.994214 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:53.994186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:54.003469 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.003443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghmg\" (UniqueName: \"kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg\") pod \"payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:54.043937 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.043732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:54.081226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.081187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:54.270041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.270017 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:43:54.272721 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:54.272696 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f26c61_45dc_422e_8cfe_4d3c27a2caa5.slice/crio-6e0cf91450356773965e37c3009d6bb6221292e769fbe795432e932cbbec745e WatchSource:0}: Error finding container 6e0cf91450356773965e37c3009d6bb6221292e769fbe795432e932cbbec745e: Status 404 returned error can't find the container with id 6e0cf91450356773965e37c3009d6bb6221292e769fbe795432e932cbbec745e Sep 10 13:43:54.293797 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.293762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" event={"ID":"00f26c61-45dc-422e-8cfe-4d3c27a2caa5","Type":"ContainerStarted","Data":"6e0cf91450356773965e37c3009d6bb6221292e769fbe795432e932cbbec745e"} Sep 10 13:43:54.329827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.329789 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:54.376540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.376505 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:43:54.377464 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:54.377421 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8955ce78_51f8_44fe_9582_ce2de64dccca.slice/crio-71aaa6ad9d8a124ddbda0e18e18534559ca320f7346c15838e6f7c287e8ec55a WatchSource:0}: Error finding container 71aaa6ad9d8a124ddbda0e18e18534559ca320f7346c15838e6f7c287e8ec55a: Status 404 returned error can't find the container with id 71aaa6ad9d8a124ddbda0e18e18534559ca320f7346c15838e6f7c287e8ec55a Sep 10 13:43:54.431426 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.431395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:54.712872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.712728 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:43:54.716966 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.716943 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.720449 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.720133 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e6d037-openshift-default-dockercfg-qdgc4\"" Sep 10 13:43:54.732902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.732633 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:43:54.755187 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.755144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:54.800454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800651 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800497 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800651 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk24g\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800651 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800651 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800697 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.800848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.800772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.901767 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.901990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.902021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk24g\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.902069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.902102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.902676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.902575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.903139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.903107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.903610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.903582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.903891 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.903868 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.903954 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.903895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.906457 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.906430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.906671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.906648 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.918385 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.918238 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:54.918686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:54.918635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk24g\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g\") pod \"e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:55.034424 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.034392 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:55.096960 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.096827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:55.217544 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.217503 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:43:55.217703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.217575 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:43:55.219160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.219122 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:55.300721 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.300642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:43:55.303418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.303385 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" event={"ID":"8955ce78-51f8-44fe-9582-ce2de64dccca","Type":"ContainerStarted","Data":"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064"} Sep 10 13:43:55.303566 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.303444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" event={"ID":"8955ce78-51f8-44fe-9582-ce2de64dccca","Type":"ContainerStarted","Data":"71aaa6ad9d8a124ddbda0e18e18534559ca320f7346c15838e6f7c287e8ec55a"} Sep 10 13:43:55.303630 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.303615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:43:55.305293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.305268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" event={"ID":"00f26c61-45dc-422e-8cfe-4d3c27a2caa5","Type":"ContainerStarted","Data":"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25"} Sep 10 13:43:55.305461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.305409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:43:55.327297 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.327249 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" podStartSLOduration=2.327233949 podStartE2EDuration="2.327233949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:55.325507684 +0000 UTC m=+810.935292598" watchObservedRunningTime="2026-09-10 13:43:55.327233949 +0000 UTC m=+810.937018861" Sep 10 13:43:55.337114 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.337088 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:43:55.340216 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:43:55.340187 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb327f38b_8381_4416_b35a_fe179cdaa842.slice/crio-15440cef2230f41fe56a2c4683dbbdc5709cdf6a0f3f0dc96044c8ceb56cd900 WatchSource:0}: Error finding container 15440cef2230f41fe56a2c4683dbbdc5709cdf6a0f3f0dc96044c8ceb56cd900: Status 404 returned error can't find the container with id 15440cef2230f41fe56a2c4683dbbdc5709cdf6a0f3f0dc96044c8ceb56cd900 Sep 10 13:43:55.343131 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.343093 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:55.343220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.343175 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:55.343256 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.343221 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:55.351902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:55.351856 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" podStartSLOduration=2.351841869 podStartE2EDuration="2.351841869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:55.34871716 +0000 UTC m=+810.958502073" watchObservedRunningTime="2026-09-10 13:43:55.351841869 +0000 UTC m=+810.961626788" Sep 10 13:43:56.160439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:56.160404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:56.316450 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:56.316411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerStarted","Data":"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365"} Sep 10 13:43:56.316634 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:56.316466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerStarted","Data":"15440cef2230f41fe56a2c4683dbbdc5709cdf6a0f3f0dc96044c8ceb56cd900"} Sep 10 13:43:56.347251 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:56.347187 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podStartSLOduration=2.347164779 podStartE2EDuration="2.347164779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:56.343089479 +0000 UTC m=+811.952874392" watchObservedRunningTime="2026-09-10 13:43:56.347164779 +0000 UTC m=+811.956949695" Sep 10 13:43:57.034763 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.034735 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:57.042474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.042445 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:57.112977 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.112941 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:57.309476 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.309443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:57.324467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.324441 2559 generic.go:356] "Generic (PLEG): container finished" podID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerID="b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e" exitCode=0 Sep 10 13:43:57.324567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.324513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerDied","Data":"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e"} Sep 10 13:43:57.324828 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.324808 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:57.325107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.325087 2559 scope.go:117] "RemoveContainer" containerID="b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e" Sep 10 13:43:57.325942 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.325658 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:43:57.326240 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.326216 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:57.326293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.326280 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:57.326329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.326312 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:43:57.438910 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.438826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:57.712665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:57.712584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:58.213034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:58.212996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:58.352391 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:58.351280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerStarted","Data":"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289"} Sep 10 13:43:58.704491 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:58.704449 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:43:58.704671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:58.704631 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:43:58.704724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:58.704682 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:43:59.665762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:59.665725 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:43:59.704587 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:59.704548 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:43:59.704749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:59.704620 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:43:59.766300 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:43:59.766257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:00.575318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:00.575276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:00.665917 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:00.665868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:00.703971 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:00.703934 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:44:00.704139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:00.704009 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:44:01.704023 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:01.703990 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:44:01.704439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:01.704060 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:44:01.937523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:01.937432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:02.067394 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:02.060966 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:02.308072 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:02.307998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:02.704409 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:02.704351 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:44:02.704779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:02.704426 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:44:03.148092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:03.148044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:03.704484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:03.704449 2559 patch_prober.go:28] interesting pod/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 10 13:44:03.704884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:03.704515 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 10 13:44:03.951647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:03.951589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:04.044561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.044524 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s"] Sep 10 13:44:04.049547 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.049515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:04.052798 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.052776 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:44:04.057331 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.057303 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:04.090835 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.090783 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s"] Sep 10 13:44:04.214851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.214806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9a01385-nj98s\" (UID: \"a9159755-57e9-4c61-a474-ce64e37769c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:04.274518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.274477 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:44:04.274682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.274538 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:44:04.315757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.315678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9a01385-nj98s\" (UID: \"a9159755-57e9-4c61-a474-ce64e37769c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:04.316522 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.316494 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e9a01385-nj98s\" (UID: \"a9159755-57e9-4c61-a474-ce64e37769c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:04.364858 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.364826 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:04.460983 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.460949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:04.661876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.661784 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s"] Sep 10 13:44:04.668354 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:44:04.668313 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9159755_57e9_4c61_a474_ce64e37769c3.slice/crio-a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2 WatchSource:0}: Error finding container a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2: Status 404 returned error can't find the container with id a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2 Sep 10 13:44:04.708049 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.708023 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:44:04.708413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.708395 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:44:04.709268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.709247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:44:04.767816 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.767771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:04.865897 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:04.865864 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:05.133693 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:05.133653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:05.261247 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:05.261210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:05.409472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:05.409389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" event={"ID":"a9159755-57e9-4c61-a474-ce64e37769c3","Type":"ContainerStarted","Data":"a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2"} Sep 10 13:44:06.319033 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.319002 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:44:06.319617 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.319088 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:44:06.419530 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.419492 2559 generic.go:356] "Generic (PLEG): container finished" podID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerID="4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05" exitCode=0 Sep 10 13:44:06.419715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.419568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerDied","Data":"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05"} Sep 10 13:44:06.420304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.420278 2559 scope.go:117] "RemoveContainer" containerID="4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05" Sep 10 13:44:06.421607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.421575 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:06.421766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.421641 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:06.421766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.421675 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:06.510447 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:06.510415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:07.452389 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:07.452017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerStarted","Data":"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b"} Sep 10 13:44:07.562000 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:07.561955 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:07.999251 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:07.999216 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:07.999489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:07.999341 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 13:44:07.999489 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:07.999404 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 13:44:08.322685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.322633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:08.459944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.459895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" event={"ID":"a9159755-57e9-4c61-a474-ce64e37769c3","Type":"ContainerStarted","Data":"b5787cec3989f6f761d520134acc82a8a190eaabebe449b323ee289a85b50712"} Sep 10 13:44:08.486943 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.486895 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" podStartSLOduration=1.558548255 podStartE2EDuration="4.486878398s" totalImagesPullingTime="2.928330143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:08.486404522 +0000 UTC m=+824.096189436" watchObservedRunningTime="2026-09-10 13:44:08.486878398 +0000 UTC m=+824.096663310" Sep 10 13:44:08.557411 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.557376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:08.658166 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.658074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:08.999873 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.999799 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 13:44:08.999873 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:08.999852 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 13:44:09.467090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:09.467051 2559 generic.go:356] "Generic (PLEG): container finished" podID="a9159755-57e9-4c61-a474-ce64e37769c3" containerID="b5787cec3989f6f761d520134acc82a8a190eaabebe449b323ee289a85b50712" exitCode=0 Sep 10 13:44:09.467500 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:09.467099 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" event={"ID":"a9159755-57e9-4c61-a474-ce64e37769c3","Type":"ContainerDied","Data":"b5787cec3989f6f761d520134acc82a8a190eaabebe449b323ee289a85b50712"} Sep 10 13:44:09.600541 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:09.600504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:09.999824 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:09.999788 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 13:44:09.999992 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:09.999842 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 13:44:10.625806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.625783 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:10.679355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.679319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca\") pod \"a9159755-57e9-4c61-a474-ce64e37769c3\" (UID: \"a9159755-57e9-4c61-a474-ce64e37769c3\") " Sep 10 13:44:10.679728 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.679700 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9159755-57e9-4c61-a474-ce64e37769c3" (UID: "a9159755-57e9-4c61-a474-ce64e37769c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:10.780561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.780534 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9159755-57e9-4c61-a474-ce64e37769c3-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:10.841014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.840928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:10.999535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.999500 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 13:44:10.999685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:10.999557 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 13:44:11.011671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.011643 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:11.169510 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.169422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:11.273165 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.273134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:11.481758 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.481688 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" Sep 10 13:44:11.481758 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.481723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s" event={"ID":"a9159755-57e9-4c61-a474-ce64e37769c3","Type":"ContainerDied","Data":"a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2"} Sep 10 13:44:11.481931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.481771 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5365f130f4438cddf8f6556e3cd2a05c959bfdfec93b1eb395324f08639c3c2" Sep 10 13:44:11.909079 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.909046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:11.999528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.999493 2559 patch_prober.go:28] interesting pod/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 10 13:44:11.999686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:11.999552 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 10 13:44:13.003876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:13.003844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:13.004230 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:13.004131 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:13.004911 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:13.004891 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:14.023630 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:14.023585 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.008322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.008283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.110481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.110448 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.229398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.229346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.408751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.408666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.563996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.563964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:15.586638 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.586608 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s"] Sep 10 13:44:15.588986 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.588966 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e9a01385-nj98s"] Sep 10 13:44:15.816757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.816716 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:44:15.816937 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.816912 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" containerID="cri-o://680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b" gracePeriod=30 Sep 10 13:44:15.842073 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:15.842052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.253774 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.253691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.396716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.396667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.479784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.479744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.619355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.619329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.800687 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.800660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:44:16.919483 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.919416 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9159755-57e9-4c61-a474-ce64e37769c3" path="/var/lib/kubelet/pods/a9159755-57e9-4c61-a474-ce64e37769c3/volumes" Sep 10 13:44:16.952025 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.951999 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:16.952611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.952595 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9159755-57e9-4c61-a474-ce64e37769c3" containerName="revoke-keys" Sep 10 13:44:16.952664 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.952614 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9159755-57e9-4c61-a474-ce64e37769c3" containerName="revoke-keys" Sep 10 13:44:16.952744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.952733 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9159755-57e9-4c61-a474-ce64e37769c3" containerName="revoke-keys" Sep 10 13:44:16.956759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.956719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:16.957395 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.957353 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:16.978307 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:16.978285 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b1d82512-gw-openshift-default-dockercfg-hqcnk\"" Sep 10 13:44:17.042057 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99xn\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042182 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.042348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.042340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.143976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.143940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144219 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144219 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99xn\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144751 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144717 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.144908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.144756 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.146534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.146500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.146694 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.146673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.233525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.233468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.233871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.233852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99xn\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn\") pod \"e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.267550 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.267528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:17.334187 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.334153 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:17.805403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.805308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:17.856888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.856852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:17.862166 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:44:17.862138 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b73a7eb_3dbb_40ef_a3a9_6bc6e0c4372e.slice/crio-a6da860c688f0eb1683267417583863b04b37a388895442f07dbf13d964cd1a6 WatchSource:0}: Error finding container a6da860c688f0eb1683267417583863b04b37a388895442f07dbf13d964cd1a6: Status 404 returned error can't find the container with id a6da860c688f0eb1683267417583863b04b37a388895442f07dbf13d964cd1a6 Sep 10 13:44:17.864348 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.864320 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:17.864425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.864400 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:17.864462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:17.864441 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:18.524549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:18.524507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerStarted","Data":"1e6948aef0cfceabcb023c55c8d39e3e0db0b0bfa226213a5b2843905f4f8f8a"} Sep 10 13:44:18.524988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:18.524566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerStarted","Data":"a6da860c688f0eb1683267417583863b04b37a388895442f07dbf13d964cd1a6"} Sep 10 13:44:18.609628 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:18.609572 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podStartSLOduration=2.609555871 podStartE2EDuration="2.609555871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:18.606171132 +0000 UTC m=+834.215956045" watchObservedRunningTime="2026-09-10 13:44:18.609555871 +0000 UTC m=+834.219340784" Sep 10 13:44:18.726133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:18.726093 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:19.268505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:19.268472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:19.273290 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:19.273263 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:19.533572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:19.533484 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:19.539864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:19.539832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:20.394482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:20.394440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:20.548904 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:20.548867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:21.105838 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.105811 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:21.191754 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191727 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.191926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191765 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.191926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191796 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.191926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.191926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191859 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.191926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191879 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191944 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.191992 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192043 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz479\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479\") pod \"a873df74-dfcd-4e31-afd6-06b372fd9d79\" (UID: \"a873df74-dfcd-4e31-afd6-06b372fd9d79\") " Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192146 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.192195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192170 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:21.192511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192392 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a873df74-dfcd-4e31-afd6-06b372fd9d79-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.192511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192398 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.192511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192409 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.192511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192429 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data" (OuterVolumeSpecName: "istio-data") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.192511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.192455 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.194208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.194179 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:21.194317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.194202 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token" (OuterVolumeSpecName: "istio-token") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:21.194317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.194221 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479" (OuterVolumeSpecName: "kube-api-access-zz479") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "kube-api-access-zz479". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:21.194317 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.194289 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a873df74-dfcd-4e31-afd6-06b372fd9d79" (UID: "a873df74-dfcd-4e31-afd6-06b372fd9d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:44:21.293332 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293303 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz479\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-kube-api-access-zz479\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.293332 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293329 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.293332 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293339 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.293578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293349 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.293578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293358 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.293578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.293381 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a873df74-dfcd-4e31-afd6-06b372fd9d79-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:21.554795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.554760 2559 generic.go:356] "Generic (PLEG): container finished" podID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerID="680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b" exitCode=0 Sep 10 13:44:21.555207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.554829 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" Sep 10 13:44:21.555207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.554857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerDied","Data":"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b"} Sep 10 13:44:21.555207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.554930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc" event={"ID":"a873df74-dfcd-4e31-afd6-06b372fd9d79","Type":"ContainerDied","Data":"8b4a59586854f6e16c04a92c71a3c26578c496f06ccc4a4b086298d4cf579c27"} Sep 10 13:44:21.555207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.554961 2559 scope.go:117] "RemoveContainer" containerID="680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b" Sep 10 13:44:21.565093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.565078 2559 scope.go:117] "RemoveContainer" containerID="4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05" Sep 10 13:44:21.573240 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.573221 2559 scope.go:117] "RemoveContainer" containerID="680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b" Sep 10 13:44:21.573588 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:44:21.573563 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b\": container with ID starting with 680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b not found: ID does not exist" containerID="680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b" Sep 10 13:44:21.573657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.573589 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b"} err="failed to get container status \"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b\": rpc error: code = NotFound desc = could not find container \"680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b\": container with ID starting with 680e156002c2e8ce87c0e480cc1dba0afb1969c197b8c7a8698cfbf21c2cf77b not found: ID does not exist" Sep 10 13:44:21.573657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.573607 2559 scope.go:117] "RemoveContainer" containerID="4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05" Sep 10 13:44:21.573882 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:44:21.573865 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05\": container with ID starting with 4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05 not found: ID does not exist" containerID="4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05" Sep 10 13:44:21.573939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.573888 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05"} err="failed to get container status \"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05\": rpc error: code = NotFound desc = could not find container \"4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05\": container with ID starting with 4c93c00807e01edc74d2c7a469a966711d731f07abf05626e8bd43fefe0ccf05 not found: ID does not exist" Sep 10 13:44:21.598274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.598252 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:44:21.640008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:21.639979 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e9a01385-openshift-default-867b4c6f6d-h82tc"] Sep 10 13:44:22.919532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:22.919491 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" path="/var/lib/kubelet/pods/a873df74-dfcd-4e31-afd6-06b372fd9d79/volumes" Sep 10 13:44:23.690124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:23.690079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:24.358464 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:24.358423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:24.480020 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:24.479963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:25.421485 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:25.421452 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:25.566761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:25.566724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:25.952599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:25.952567 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:26.730685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:26.730658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:26.861487 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:26.861412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:27.325754 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.325721 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:27.325906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.325771 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:27.352857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.352825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:27.595752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.595667 2559 generic.go:356] "Generic (PLEG): container finished" podID="b327f38b-8381-4416-b35a-fe179cdaa842" containerID="2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365" exitCode=0 Sep 10 13:44:27.595752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.595734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerDied","Data":"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365"} Sep 10 13:44:27.596175 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.596159 2559 scope.go:117] "RemoveContainer" containerID="2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365" Sep 10 13:44:27.597330 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.597297 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:27.597488 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.597397 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:27.597488 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.597449 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:27.871000 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:27.870891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:28.604130 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:28.604093 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerStarted","Data":"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17"} Sep 10 13:44:28.822855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:28.822763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:29.034682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:29.034650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:44:29.035506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:29.035478 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:29.035716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:29.035692 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:29.334703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:29.334619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:30.035169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:30.035125 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:30.035742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:30.035193 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:30.796991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:30.796961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:31.035179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:31.035142 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:31.035673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:31.035219 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:31.200263 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:31.200175 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:44:32.035304 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:32.035265 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:32.036314 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:32.035904 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:32.697482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:32.697431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:33.035945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:33.035477 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 10 13:44:33.035945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:33.035559 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 10 13:44:34.055981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:34.055943 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:44:34.056475 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:34.056408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:44:34.057040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:34.057003 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:44:34.256515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:34.256477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:36.206322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:36.206278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:37.240196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:37.240155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:37.477265 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:37.477234 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:38.783945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:38.783908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:40.110877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.110836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:40.129988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.129955 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9"] Sep 10 13:44:40.130381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.130347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:40.130381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.130361 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:40.130560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.130399 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:40.130560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.130405 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:40.130560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.130471 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:40.135294 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.135272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:40.140592 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.140567 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:44:40.143753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.143523 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9"] Sep 10 13:44:40.288736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.288702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9\" (UID: \"896beaec-072c-40d2-9424-a4262bd60e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:40.389510 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.389426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9\" (UID: \"896beaec-072c-40d2-9424-a4262bd60e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:40.390067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.390041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9\" (UID: \"896beaec-072c-40d2-9424-a4262bd60e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:40.448337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.448317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:40.745344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:40.745302 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9"] Sep 10 13:44:40.746837 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:44:40.746809 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896beaec_072c_40d2_9424_a4262bd60e15.slice/crio-c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7 WatchSource:0}: Error finding container c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7: Status 404 returned error can't find the container with id c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7 Sep 10 13:44:41.316166 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.316103 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:41.695575 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.695548 2559 generic.go:356] "Generic (PLEG): container finished" podID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerID="1e6948aef0cfceabcb023c55c8d39e3e0db0b0bfa226213a5b2843905f4f8f8a" exitCode=0 Sep 10 13:44:41.695680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.695617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerDied","Data":"1e6948aef0cfceabcb023c55c8d39e3e0db0b0bfa226213a5b2843905f4f8f8a"} Sep 10 13:44:41.696068 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.696047 2559 scope.go:117] "RemoveContainer" containerID="1e6948aef0cfceabcb023c55c8d39e3e0db0b0bfa226213a5b2843905f4f8f8a" Sep 10 13:44:41.697190 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697160 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:41.697299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697246 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:41.697393 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697296 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:41.697461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697441 2559 generic.go:356] "Generic (PLEG): container finished" podID="896beaec-072c-40d2-9424-a4262bd60e15" containerID="62181f33d63b54f8d0ff08382542d209e0e07bc510216041905144537a476ce2" exitCode=0 Sep 10 13:44:41.697526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" event={"ID":"896beaec-072c-40d2-9424-a4262bd60e15","Type":"ContainerDied","Data":"62181f33d63b54f8d0ff08382542d209e0e07bc510216041905144537a476ce2"} Sep 10 13:44:41.697565 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:41.697550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" event={"ID":"896beaec-072c-40d2-9424-a4262bd60e15","Type":"ContainerStarted","Data":"c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7"} Sep 10 13:44:42.705514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:42.705481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerStarted","Data":"81ce303222f4f61b9b22d20f0b702f59543d5aa7659dd3f9c6f2b502ecb61152"} Sep 10 13:44:42.860190 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:42.860142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:42.884689 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:42.884646 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:43.016740 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.016705 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca\") pod \"896beaec-072c-40d2-9424-a4262bd60e15\" (UID: \"896beaec-072c-40d2-9424-a4262bd60e15\") " Sep 10 13:44:43.017079 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.017052 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "896beaec-072c-40d2-9424-a4262bd60e15" (UID: "896beaec-072c-40d2-9424-a4262bd60e15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:43.118016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.117986 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/896beaec-072c-40d2-9424-a4262bd60e15-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:43.268081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.268002 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:43.268244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.268217 2559 patch_prober.go:28] interesting pod/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:44:43.268306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.268282 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 13:44:43.712826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.712751 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" Sep 10 13:44:43.712826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.712788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9" event={"ID":"896beaec-072c-40d2-9424-a4262bd60e15","Type":"ContainerDied","Data":"c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7"} Sep 10 13:44:43.713224 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.712832 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26817ddc08db38882d8f68579745cb66def80454ab0e9cbf896490d8cb795d7" Sep 10 13:44:43.993140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:43.993043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:44.267854 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:44.267819 2559 patch_prober.go:28] interesting pod/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:44:44.268018 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:44.267868 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 13:44:45.089503 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:45.089469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:45.268701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:45.268667 2559 patch_prober.go:28] interesting pod/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:44:45.268871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:45.268717 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 13:44:46.034736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:46.034688 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:44:46.268068 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:46.268033 2559 patch_prober.go:28] interesting pod/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:44:46.268432 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:46.268088 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 13:44:46.318280 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:46.318196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:47.268752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:47.268715 2559 patch_prober.go:28] interesting pod/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 10 13:44:47.269128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:47.268774 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 10 13:44:47.692850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:47.692760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:48.272579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:48.272544 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:48.273017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:48.272826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:48.273673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:48.273655 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:50.960943 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:50.960911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:51.090206 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.090160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:51.277961 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.277909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:51.390353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.390323 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9"] Sep 10 13:44:51.392833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.392813 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1d82512-7xch9"] Sep 10 13:44:51.412842 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.412814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:51.595582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.595492 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:51.595744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:51.595718 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" containerID="cri-o://81ce303222f4f61b9b22d20f0b702f59543d5aa7659dd3f9c6f2b502ecb61152" gracePeriod=30 Sep 10 13:44:52.311119 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.311089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:52.411962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.411907 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:52.456119 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.456091 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:44:52.456631 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.456611 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="896beaec-072c-40d2-9424-a4262bd60e15" containerName="revoke-keys" Sep 10 13:44:52.456631 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.456632 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="896beaec-072c-40d2-9424-a4262bd60e15" containerName="revoke-keys" Sep 10 13:44:52.456858 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.456730 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a873df74-dfcd-4e31-afd6-06b372fd9d79" containerName="istio-proxy" Sep 10 13:44:52.456858 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.456746 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="896beaec-072c-40d2-9424-a4262bd60e15" containerName="revoke-keys" Sep 10 13:44:52.459920 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.459899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.463987 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.463968 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d6a65c4-openshift-default-dockercfg-qclpt\"" Sep 10 13:44:52.488617 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.488592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:44:52.505225 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.505184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:52.606591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606513 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lqr\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606632 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606806 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606826 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.606939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.606862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lqr\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708725 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.708988 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.708905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.709167 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.709148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.710691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.710668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.710975 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.710956 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.718450 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.718416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.719181 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.719157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lqr\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr\") pod \"e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.729701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.729667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:52.770560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.770540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:52.919603 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:52.919533 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="896beaec-072c-40d2-9424-a4262bd60e15" path="/var/lib/kubelet/pods/896beaec-072c-40d2-9424-a4262bd60e15/volumes" Sep 10 13:44:53.062121 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.062092 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:44:53.062952 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:44:53.062919 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf29c98a_8f2d_43e2_b7b8_60e8ee24ad6f.slice/crio-1ced96e6bf291b7c1a0daf669cd58cf3cc4729eae92665127bf71b6da54e005b WatchSource:0}: Error finding container 1ced96e6bf291b7c1a0daf669cd58cf3cc4729eae92665127bf71b6da54e005b: Status 404 returned error can't find the container with id 1ced96e6bf291b7c1a0daf669cd58cf3cc4729eae92665127bf71b6da54e005b Sep 10 13:44:53.065201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.065169 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:53.065281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.065234 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:53.065281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.065260 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:44:53.196053 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.195961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:53.797528 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.797489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerStarted","Data":"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7"} Sep 10 13:44:53.797995 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.797550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerStarted","Data":"1ced96e6bf291b7c1a0daf669cd58cf3cc4729eae92665127bf71b6da54e005b"} Sep 10 13:44:53.824912 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:53.824866 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podStartSLOduration=1.82485115 podStartE2EDuration="1.82485115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:53.823428412 +0000 UTC m=+869.433213325" watchObservedRunningTime="2026-09-10 13:44:53.82485115 +0000 UTC m=+869.434636063" Sep 10 13:44:54.412787 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:54.412752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:54.771193 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:54.771163 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:54.776018 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:54.775994 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:54.805850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:54.805822 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:54.806678 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:54.806657 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:44:55.191775 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:55.191697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:55.293894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:55.293859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:55.524007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:55.523974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:56.795658 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.795542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:56.821379 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.821334 2559 generic.go:356] "Generic (PLEG): container finished" podID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerID="81ce303222f4f61b9b22d20f0b702f59543d5aa7659dd3f9c6f2b502ecb61152" exitCode=0 Sep 10 13:44:56.821543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.821395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerDied","Data":"81ce303222f4f61b9b22d20f0b702f59543d5aa7659dd3f9c6f2b502ecb61152"} Sep 10 13:44:56.821543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.821470 2559 scope.go:117] "RemoveContainer" containerID="1e6948aef0cfceabcb023c55c8d39e3e0db0b0bfa226213a5b2843905f4f8f8a" Sep 10 13:44:56.870101 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.870051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:56.895497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.895474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:56.953995 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.953969 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954029 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954065 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954115 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954141 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b99xn\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954231 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:56.954494 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954312 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954496 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:56.954549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954516 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:56.954661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:44:56.954715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954673 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket\") pod \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\" (UID: \"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e\") " Sep 10 13:44:56.954942 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.954915 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:56.955311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.955070 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:56.955311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.955092 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:56.955311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.955106 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:56.955311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.955120 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:56.955311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.955134 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:56.956573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.956551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:44:56.956683 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.956588 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:44:56.956683 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.956600 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:56.956771 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.956693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn" (OuterVolumeSpecName: "kube-api-access-b99xn") pod "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" (UID: "8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e"). InnerVolumeSpecName "kube-api-access-b99xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:44:56.996763 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:56.996740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:57.055685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.055657 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:57.055685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.055682 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:57.055685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.055693 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99xn\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-kube-api-access-b99xn\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:57.055906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.055701 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:44:57.372481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.372449 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:57.469132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.469100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:57.828415 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.828356 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" Sep 10 13:44:57.828415 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.828397 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64" event={"ID":"8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e","Type":"ContainerDied","Data":"a6da860c688f0eb1683267417583863b04b37a388895442f07dbf13d964cd1a6"} Sep 10 13:44:57.828931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.828453 2559 scope.go:117] "RemoveContainer" containerID="81ce303222f4f61b9b22d20f0b702f59543d5aa7659dd3f9c6f2b502ecb61152" Sep 10 13:44:57.878207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.878180 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:57.886112 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.886090 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b1d82512-gw-openshift-default-5cc86db8b8-4rd64"] Sep 10 13:44:57.944074 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:57.944045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:58.917851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:58.917819 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" path="/var/lib/kubelet/pods/8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e/volumes" Sep 10 13:44:59.051333 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:59.051300 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:44:59.935939 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:44:59.935895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:00.142453 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142423 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:00.142862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142847 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.142862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142862 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.142963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142880 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.142963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142886 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.142963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142945 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.142963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.142952 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b73a7eb-3dbb-40ef-a3a9-6bc6e0c4372e" containerName="istio-proxy" Sep 10 13:45:00.147755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.147733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" Sep 10 13:45:00.149670 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.149648 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n"] Sep 10 13:45:00.150277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.150258 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:45:00.153225 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.153208 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" Sep 10 13:45:00.153292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.153270 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr"] Sep 10 13:45:00.157535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.157515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" Sep 10 13:45:00.157741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.157598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:00.162825 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.162807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" Sep 10 13:45:00.165561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.165543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr"] Sep 10 13:45:00.169089 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.169061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n"] Sep 10 13:45:00.176720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.176702 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" Sep 10 13:45:00.214384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.214351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" Sep 10 13:45:00.449576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.449512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:00.452216 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:00.452190 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08403753_6e22_4a86_8626_a0ed089b8f0a.slice/crio-df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf WatchSource:0}: Error finding container df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf: Status 404 returned error can't find the container with id df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf Sep 10 13:45:00.469852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.469832 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n"] Sep 10 13:45:00.470998 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:00.470975 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e1dcc5_1a75_435f_a18e_1fae156c13ef.slice/crio-24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba WatchSource:0}: Error finding container 24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba: Status 404 returned error can't find the container with id 24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba Sep 10 13:45:00.515874 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.515855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr"] Sep 10 13:45:00.517271 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:00.517249 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f739f0_9b14_42f7_8ad7_67cd3a5b3668.slice/crio-f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a WatchSource:0}: Error finding container f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a: Status 404 returned error can't find the container with id f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a Sep 10 13:45:00.844734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.844710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:00.852475 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.852451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerStarted","Data":"bd725afb0956dd732f5e6f4c98347a6b2511dbe71cfc8c8c4da7d0c161094548"} Sep 10 13:45:00.852573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.852510 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerStarted","Data":"df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf"} Sep 10 13:45:00.854111 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.854082 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" event={"ID":"93f739f0-9b14-42f7-8ad7-67cd3a5b3668","Type":"ContainerStarted","Data":"a4c1a13180b601acf4c1dc897a3c45386cbba4824c8b2417de83e5668a7af97a"} Sep 10 13:45:00.854223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.854136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" event={"ID":"93f739f0-9b14-42f7-8ad7-67cd3a5b3668","Type":"ContainerStarted","Data":"f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a"} Sep 10 13:45:00.855730 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.855710 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" event={"ID":"d0e1dcc5-1a75-435f-a18e-1fae156c13ef","Type":"ContainerStarted","Data":"49fffcc4c070312d631b0bae3974a38f95233aedbf72761b72cb4333bc249a47"} Sep 10 13:45:00.855822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.855748 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" event={"ID":"d0e1dcc5-1a75-435f-a18e-1fae156c13ef","Type":"ContainerStarted","Data":"24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba"} Sep 10 13:45:00.873212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.873171 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" podStartSLOduration=0.873158179 podStartE2EDuration="873.158179ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:00.871288934 +0000 UTC m=+876.481073844" watchObservedRunningTime="2026-09-10 13:45:00.873158179 +0000 UTC m=+876.482943114" Sep 10 13:45:00.890415 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.890349 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" podStartSLOduration=0.890332389 podStartE2EDuration="890.332389ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:00.886950538 +0000 UTC m=+876.496735467" watchObservedRunningTime="2026-09-10 13:45:00.890332389 +0000 UTC m=+876.500117304" Sep 10 13:45:00.904016 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:00.903966 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" podStartSLOduration=0.903951781 podStartE2EDuration="903.951781ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:00.902017786 +0000 UTC m=+876.511802701" watchObservedRunningTime="2026-09-10 13:45:00.903951781 +0000 UTC m=+876.513736694" Sep 10 13:45:01.088728 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.088701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:45:01.868233 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.868196 2559 generic.go:356] "Generic (PLEG): container finished" podID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerID="bd725afb0956dd732f5e6f4c98347a6b2511dbe71cfc8c8c4da7d0c161094548" exitCode=7 Sep 10 13:45:01.868466 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.868287 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerDied","Data":"bd725afb0956dd732f5e6f4c98347a6b2511dbe71cfc8c8c4da7d0c161094548"} Sep 10 13:45:01.868532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.868481 2559 scope.go:117] "RemoveContainer" containerID="bd725afb0956dd732f5e6f4c98347a6b2511dbe71cfc8c8c4da7d0c161094548" Sep 10 13:45:01.872107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.872080 2559 generic.go:356] "Generic (PLEG): container finished" podID="93f739f0-9b14-42f7-8ad7-67cd3a5b3668" containerID="a4c1a13180b601acf4c1dc897a3c45386cbba4824c8b2417de83e5668a7af97a" exitCode=0 Sep 10 13:45:01.872709 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.872491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" event={"ID":"93f739f0-9b14-42f7-8ad7-67cd3a5b3668","Type":"ContainerDied","Data":"a4c1a13180b601acf4c1dc897a3c45386cbba4824c8b2417de83e5668a7af97a"} Sep 10 13:45:01.875115 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.875090 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0e1dcc5-1a75-435f-a18e-1fae156c13ef" containerID="49fffcc4c070312d631b0bae3974a38f95233aedbf72761b72cb4333bc249a47" exitCode=0 Sep 10 13:45:01.875227 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:01.875148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" event={"ID":"d0e1dcc5-1a75-435f-a18e-1fae156c13ef","Type":"ContainerDied","Data":"49fffcc4c070312d631b0bae3974a38f95233aedbf72761b72cb4333bc249a47"} Sep 10 13:45:02.883637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:02.883537 2559 generic.go:356] "Generic (PLEG): container finished" podID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerID="a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5" exitCode=7 Sep 10 13:45:02.884099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:02.883627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerDied","Data":"a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5"} Sep 10 13:45:02.884099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:02.883733 2559 scope.go:117] "RemoveContainer" containerID="bd725afb0956dd732f5e6f4c98347a6b2511dbe71cfc8c8c4da7d0c161094548" Sep 10 13:45:02.884099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:02.883789 2559 scope.go:117] "RemoveContainer" containerID="a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5" Sep 10 13:45:02.884099 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:45:02.884073 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982_odh-ai-gateway-infra(08403753-6e22-4a86-8626-a0ed089b8f0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" Sep 10 13:45:03.062637 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.062609 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" Sep 10 13:45:03.105521 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.105487 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" Sep 10 13:45:03.891454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.891422 2559 scope.go:117] "RemoveContainer" containerID="a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5" Sep 10 13:45:03.891903 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:45:03.891690 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982_odh-ai-gateway-infra(08403753-6e22-4a86-8626-a0ed089b8f0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" Sep 10 13:45:03.892562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.892545 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" Sep 10 13:45:03.892665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.892568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr" event={"ID":"93f739f0-9b14-42f7-8ad7-67cd3a5b3668","Type":"ContainerDied","Data":"f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a"} Sep 10 13:45:03.892665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.892609 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90df3ed336a20cca49606f8558893dc1d6c4306e797ef5e910fcec4f8c4e71a" Sep 10 13:45:03.894060 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.894040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" event={"ID":"d0e1dcc5-1a75-435f-a18e-1fae156c13ef","Type":"ContainerDied","Data":"24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba"} Sep 10 13:45:03.894145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.894073 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24bc1c47c0b393cf10816c263b7bc21bd0f35455ed6e06ee24688f261a89f7ba" Sep 10 13:45:03.894145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:03.894045 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n" Sep 10 13:45:10.846515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:10.846479 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:45:10.846931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:10.846534 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:45:11.947754 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.947720 2559 generic.go:356] "Generic (PLEG): container finished" podID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerID="f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7" exitCode=0 Sep 10 13:45:11.948212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.947797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerDied","Data":"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7"} Sep 10 13:45:11.948212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.948181 2559 scope.go:117] "RemoveContainer" containerID="f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7" Sep 10 13:45:11.949426 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.949390 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:11.949496 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.949483 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:11.949534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:11.949519 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:12.958140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:12.958108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerStarted","Data":"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10"} Sep 10 13:45:13.770895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:13.770862 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:13.771142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:13.771057 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:45:13.771142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:13.771110 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:45:14.771316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:14.771284 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:45:14.771725 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:14.771337 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:45:15.771771 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:15.771737 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:45:15.772161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:15.771792 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:45:15.912432 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:15.912404 2559 scope.go:117] "RemoveContainer" containerID="a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5" Sep 10 13:45:16.403737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:16.403660 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:16.771536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:16.771506 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:45:16.771695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:16.771560 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:45:16.983581 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:16.983551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerStarted","Data":"de7c0718e755c1610a5b7a3a646e8701ada900add0ff48c4b076800d0c6de8a9"} Sep 10 13:45:17.771295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:17.771262 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 10 13:45:17.771470 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:17.771329 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 10 13:45:17.992088 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:17.992053 2559 generic.go:356] "Generic (PLEG): container finished" podID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerID="de7c0718e755c1610a5b7a3a646e8701ada900add0ff48c4b076800d0c6de8a9" exitCode=0 Sep 10 13:45:17.992574 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:17.992129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerDied","Data":"de7c0718e755c1610a5b7a3a646e8701ada900add0ff48c4b076800d0c6de8a9"} Sep 10 13:45:17.992574 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:17.992211 2559 scope.go:117] "RemoveContainer" containerID="a180aab982eff166e140be9494eb92aa7af5aa4901ce2e0f676d9855acbdb0d5" Sep 10 13:45:18.022548 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:18.022486 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:18.775540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:18.775511 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:18.775810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:18.775776 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:18.776578 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:18.776562 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:19.155515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:19.155493 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" Sep 10 13:45:19.354669 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:19.354586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:20.008173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.008145 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" Sep 10 13:45:20.008173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.008159 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982" event={"ID":"08403753-6e22-4a86-8626-a0ed089b8f0a","Type":"ContainerDied","Data":"df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf"} Sep 10 13:45:20.008402 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.008198 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6c87250239c0a47e7ad55d16f48a5ff300d96e14ec7c81a6ccc575f79b5fbf" Sep 10 13:45:20.045863 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.045839 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:20.051286 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.051266 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0d6a65c4-29817465-p4982"] Sep 10 13:45:20.227362 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.227329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:20.918927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:20.918893 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" path="/var/lib/kubelet/pods/08403753-6e22-4a86-8626-a0ed089b8f0a/volumes" Sep 10 13:45:21.178542 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178471 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc"] Sep 10 13:45:21.178878 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178865 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f739f0-9b14-42f7-8ad7-67cd3a5b3668" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178879 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f739f0-9b14-42f7-8ad7-67cd3a5b3668" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178887 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178893 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178899 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178906 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178920 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e1dcc5-1a75-435f-a18e-1fae156c13ef" containerName="cleanup" Sep 10 13:45:21.178929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178925 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e1dcc5-1a75-435f-a18e-1fae156c13ef" containerName="cleanup" Sep 10 13:45:21.179170 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178989 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.179170 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.178998 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e1dcc5-1a75-435f-a18e-1fae156c13ef" containerName="cleanup" Sep 10 13:45:21.179170 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.179004 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:21.179170 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.179010 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f739f0-9b14-42f7-8ad7-67cd3a5b3668" containerName="cleanup" Sep 10 13:45:21.182647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.182629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:21.185269 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.185249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:45:21.195793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.195772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc"] Sep 10 13:45:21.296557 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.296525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:21.304562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.304539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc\" (UID: \"5025db1d-2f3f-4ee5-a722-86e212bfcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:21.405294 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.405260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc\" (UID: \"5025db1d-2f3f-4ee5-a722-86e212bfcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:21.405902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.405881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc\" (UID: \"5025db1d-2f3f-4ee5-a722-86e212bfcf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:21.492862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.492790 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:21.802674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:21.802644 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc"] Sep 10 13:45:21.802849 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:21.802828 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5025db1d_2f3f_4ee5_a722_86e212bfcf5e.slice/crio-8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb WatchSource:0}: Error finding container 8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb: Status 404 returned error can't find the container with id 8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb Sep 10 13:45:22.020955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:22.020930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" event={"ID":"5025db1d-2f3f-4ee5-a722-86e212bfcf5e","Type":"ContainerStarted","Data":"8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb"} Sep 10 13:45:22.456429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:22.456402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:23.028402 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:23.028351 2559 generic.go:356] "Generic (PLEG): container finished" podID="5025db1d-2f3f-4ee5-a722-86e212bfcf5e" containerID="c0df0fdd4b3dba98cf3803967b6014f01d839a371a66d0897d2c88a33b670565" exitCode=0 Sep 10 13:45:23.028550 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:23.028435 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" event={"ID":"5025db1d-2f3f-4ee5-a722-86e212bfcf5e","Type":"ContainerDied","Data":"c0df0fdd4b3dba98cf3803967b6014f01d839a371a66d0897d2c88a33b670565"} Sep 10 13:45:23.227619 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:23.227586 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:24.174265 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.174239 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:24.331567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.331496 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca\") pod \"5025db1d-2f3f-4ee5-a722-86e212bfcf5e\" (UID: \"5025db1d-2f3f-4ee5-a722-86e212bfcf5e\") " Sep 10 13:45:24.331834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.331812 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5025db1d-2f3f-4ee5-a722-86e212bfcf5e" (UID: "5025db1d-2f3f-4ee5-a722-86e212bfcf5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:24.432847 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.432820 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5025db1d-2f3f-4ee5-a722-86e212bfcf5e-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:24.922360 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.922328 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:45:24.922581 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:24.922328 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:45:25.042894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:25.042856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" event={"ID":"5025db1d-2f3f-4ee5-a722-86e212bfcf5e","Type":"ContainerDied","Data":"8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb"} Sep 10 13:45:25.043042 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:25.042935 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc1ca6322594dd340fde8eee334615dac646a5bde09803bd0e7babe8b7ffdbb" Sep 10 13:45:25.043042 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:25.042880 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc" Sep 10 13:45:25.208653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:25.208570 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:45:26.016786 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:26.016747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:26.123958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:26.123794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:26.875864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:26.875826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:27.079569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.079527 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:27.223649 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.223567 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc"] Sep 10 13:45:27.230163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.230134 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d6a65c4-wthcc"] Sep 10 13:45:27.284690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.284634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:27.457222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.457187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:27.688002 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:27.687970 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:28.547715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:28.547679 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:45:28.548138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:28.547938 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" containerID="cri-o://03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10" gracePeriod=30 Sep 10 13:45:28.578228 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:28.578198 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:28.919260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:28.919167 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5025db1d-2f3f-4ee5-a722-86e212bfcf5e" path="/var/lib/kubelet/pods/5025db1d-2f3f-4ee5-a722-86e212bfcf5e/volumes" Sep 10 13:45:28.987262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:28.987217 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:29.087196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.087155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:29.327929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.327898 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:45:29.328331 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328319 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5025db1d-2f3f-4ee5-a722-86e212bfcf5e" containerName="revoke-keys" Sep 10 13:45:29.328411 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328333 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5025db1d-2f3f-4ee5-a722-86e212bfcf5e" containerName="revoke-keys" Sep 10 13:45:29.328411 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:29.328411 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328352 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:29.328516 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328457 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08403753-6e22-4a86-8626-a0ed089b8f0a" containerName="cleanup" Sep 10 13:45:29.328516 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.328468 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5025db1d-2f3f-4ee5-a722-86e212bfcf5e" containerName="revoke-keys" Sep 10 13:45:29.332822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.332800 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.336615 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.336593 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e404ac71-openshift-default-dockercfg-gfjxc\"" Sep 10 13:45:29.345355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.345327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:45:29.362534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.362494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:29.468736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.468700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:29.476902 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.476868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.476991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.476914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477033 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.476992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqll\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477033 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477024 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.477296 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.477200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.578853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.578853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqll\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.578985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579416 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.579868 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.579737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.581277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.581249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.581644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.581621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.595631 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.595605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqll\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.597398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.597375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token\") pod \"e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.645671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.645631 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:29.928946 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.928921 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:45:29.929672 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:29.929643 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bf6fac_4469_4f51_99ea_64d57623fe0d.slice/crio-f316152813d88542d10fe5de17ba0a72d8aa993028c03ff61e9c493573ee654d WatchSource:0}: Error finding container f316152813d88542d10fe5de17ba0a72d8aa993028c03ff61e9c493573ee654d: Status 404 returned error can't find the container with id f316152813d88542d10fe5de17ba0a72d8aa993028c03ff61e9c493573ee654d Sep 10 13:45:29.931955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.931919 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:29.932066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.931999 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:29.932066 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.932042 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:29.968217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:29.968193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:30.077618 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.077574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerStarted","Data":"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde"} Sep 10 13:45:30.077808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.077639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerStarted","Data":"f316152813d88542d10fe5de17ba0a72d8aa993028c03ff61e9c493573ee654d"} Sep 10 13:45:30.098146 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.098086 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podStartSLOduration=1.09806444 podStartE2EDuration="1.09806444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:30.097238509 +0000 UTC m=+905.707023428" watchObservedRunningTime="2026-09-10 13:45:30.09806444 +0000 UTC m=+905.707849353" Sep 10 13:45:30.646478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.646442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:30.647822 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.647795 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:45:30.647882 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:30.647853 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:45:31.422448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:31.422412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:31.650906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:31.650878 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:31.672941 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:31.672852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:31.844097 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:31.844063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:32.064617 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:32.064583 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:32.090096 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:32.090072 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:32.090935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:32.090913 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:33.091036 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.090993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:33.802396 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.802357 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:33.819413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7lqr\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819477 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819510 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819532 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819588 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819647 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data\") pod \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\" (UID: \"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f\") " Sep 10 13:45:33.819801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819717 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:45:33.819801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819751 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:45:33.820032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.819928 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:45:33.820084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.820030 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data" (OuterVolumeSpecName: "istio-data") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:45:33.820084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.820070 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.820538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.820089 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.820538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.820104 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.820538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.820147 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:33.821628 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.821593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token" (OuterVolumeSpecName: "istio-token") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:33.821973 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.821948 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:45:33.822082 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.822062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr" (OuterVolumeSpecName: "kube-api-access-b7lqr") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "kube-api-access-b7lqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:45:33.822870 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.822853 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" (UID: "af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:45:33.921123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921099 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.921123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921121 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.921123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921129 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.921322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921138 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.921322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921151 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:33.921322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:33.921163 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7lqr\" (UniqueName: \"kubernetes.io/projected/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f-kube-api-access-b7lqr\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:34.104251 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.104153 2559 generic.go:356] "Generic (PLEG): container finished" podID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerID="03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10" exitCode=0 Sep 10 13:45:34.104251 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.104224 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" Sep 10 13:45:34.104660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.104206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerDied","Data":"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10"} Sep 10 13:45:34.104660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.104421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" event={"ID":"af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f","Type":"ContainerDied","Data":"1ced96e6bf291b7c1a0daf669cd58cf3cc4729eae92665127bf71b6da54e005b"} Sep 10 13:45:34.104660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.104449 2559 scope.go:117] "RemoveContainer" containerID="03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10" Sep 10 13:45:34.114865 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.114836 2559 scope.go:117] "RemoveContainer" containerID="f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7" Sep 10 13:45:34.124671 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.124654 2559 scope.go:117] "RemoveContainer" containerID="03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10" Sep 10 13:45:34.124938 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:45:34.124919 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10\": container with ID starting with 03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10 not found: ID does not exist" containerID="03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10" Sep 10 13:45:34.125027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.124945 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10"} err="failed to get container status \"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10\": rpc error: code = NotFound desc = could not find container \"03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10\": container with ID starting with 03f15ae2ca417f4f56184b9add7484cf3bfde89e109b8c2be62b2ec696cddf10 not found: ID does not exist" Sep 10 13:45:34.125027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.124960 2559 scope.go:117] "RemoveContainer" containerID="f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7" Sep 10 13:45:34.125173 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:45:34.125156 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7\": container with ID starting with f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7 not found: ID does not exist" containerID="f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7" Sep 10 13:45:34.125225 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.125177 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7"} err="failed to get container status \"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7\": rpc error: code = NotFound desc = could not find container \"f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7\": container with ID starting with f5580ec13bf131fb3ffcc024f7cd9ebfc3a588740b4313e822d52a8559b475d7 not found: ID does not exist" Sep 10 13:45:34.146493 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.146468 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:45:34.156835 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.156812 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8"] Sep 10 13:45:34.229661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.229630 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:34.776631 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.776594 2559 patch_prober.go:28] interesting pod/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:45:34.776799 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.776652 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d6a65c4-openshift-default-6fd74fcc4b-29jg8" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:45:34.918656 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:34.918619 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" path="/var/lib/kubelet/pods/af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f/volumes" Sep 10 13:45:35.560216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:35.560176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:35.670667 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:35.670629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:36.039139 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.039094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:36.433892 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.433804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:36.539196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.539161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:36.616978 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.616947 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:45:36.617591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.617571 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.617690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.617593 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.617749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.617736 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.617803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.617750 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.622222 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.622200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.624727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.624707 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e404ac71\"" Sep 10 13:45:36.631567 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.631547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:45:36.648025 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.647993 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9g7\" (UniqueName: \"kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.648133 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.648035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.706644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.706582 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:45:36.707084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.707065 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.707164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.707087 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af29c98a-8f2d-43e2-b7b8-60e8ee24ad6f" containerName="istio-proxy" Sep 10 13:45:36.710292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.710273 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.713098 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.713081 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e404ac71-dockercfg-5gqvf\"" Sep 10 13:45:36.722437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.722416 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:45:36.748908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.748868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.749075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.748923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9g7\" (UniqueName: \"kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.749075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.748964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.749075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.749014 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7m9j\" (UniqueName: \"kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.750312 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.750281 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.764566 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.764532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9g7\" (UniqueName: \"kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7\") pod \"payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:36.850469 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.850439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.850626 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.850496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7m9j\" (UniqueName: \"kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.851113 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.851092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.860338 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.860309 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7m9j\" (UniqueName: \"kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j\") pod \"payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:36.934069 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:36.934039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:37.021562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:37.021530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:37.214084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:37.214052 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:45:37.214626 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:37.214602 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8714dcb2_e146_4507_82e5_64e073f4b11e.slice/crio-a0ed9f360dcfc1f5f044a7d9a8c297fb69897e4650193d9b20fd0e328d766b14 WatchSource:0}: Error finding container a0ed9f360dcfc1f5f044a7d9a8c297fb69897e4650193d9b20fd0e328d766b14: Status 404 returned error can't find the container with id a0ed9f360dcfc1f5f044a7d9a8c297fb69897e4650193d9b20fd0e328d766b14 Sep 10 13:45:37.299746 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:37.299721 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:45:37.300715 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:37.300692 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15385cf_3d88_4a9b_bacb_e4e0c981b7ac.slice/crio-827815c31f26120a8c71e1e8f31698f098adbf21548d4d0af6cb6bbaeb36c75b WatchSource:0}: Error finding container 827815c31f26120a8c71e1e8f31698f098adbf21548d4d0af6cb6bbaeb36c75b: Status 404 returned error can't find the container with id 827815c31f26120a8c71e1e8f31698f098adbf21548d4d0af6cb6bbaeb36c75b Sep 10 13:45:38.133842 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.133803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" event={"ID":"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac","Type":"ContainerStarted","Data":"7d08c1bfc4028146ed738262a6c076c1df205e04a2d5894663929280b7dc99af"} Sep 10 13:45:38.134355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.133869 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:38.134355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.133893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" event={"ID":"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac","Type":"ContainerStarted","Data":"827815c31f26120a8c71e1e8f31698f098adbf21548d4d0af6cb6bbaeb36c75b"} Sep 10 13:45:38.135497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.135465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" event={"ID":"8714dcb2-e146-4507-82e5-64e073f4b11e","Type":"ContainerStarted","Data":"582a882478c8ef87fde1cd1c5867b20922714031c1ece99e7ed54d40112a0c12"} Sep 10 13:45:38.135625 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.135513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" event={"ID":"8714dcb2-e146-4507-82e5-64e073f4b11e","Type":"ContainerStarted","Data":"a0ed9f360dcfc1f5f044a7d9a8c297fb69897e4650193d9b20fd0e328d766b14"} Sep 10 13:45:38.135625 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.135547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:38.155743 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.155699 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" podStartSLOduration=2.155685483 podStartE2EDuration="2.155685483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:38.154269368 +0000 UTC m=+913.764054285" watchObservedRunningTime="2026-09-10 13:45:38.155685483 +0000 UTC m=+913.765470395" Sep 10 13:45:38.176359 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:38.176317 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" podStartSLOduration=2.176303859 podStartE2EDuration="2.176303859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:38.175560351 +0000 UTC m=+913.785345266" watchObservedRunningTime="2026-09-10 13:45:38.176303859 +0000 UTC m=+913.786088772" Sep 10 13:45:44.918520 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:44.918483 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:45:46.244609 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:46.244576 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:48.937811 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:48.937770 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn"] Sep 10 13:45:48.942624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:48.942600 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:48.947644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:48.947617 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:45:48.966135 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:48.966109 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn"] Sep 10 13:45:49.063468 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.063430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn\" (UID: \"0839bfbb-a478-424d-97e3-1438ba3a6a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:49.145291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.145254 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:45:49.145504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.145465 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:45:49.164738 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.164711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn\" (UID: \"0839bfbb-a478-424d-97e3-1438ba3a6a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:49.165321 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.165300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn\" (UID: \"0839bfbb-a478-424d-97e3-1438ba3a6a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:49.175323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.175297 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:49.261919 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.261885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:49.580445 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:49.580418 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn"] Sep 10 13:45:49.582171 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:49.582137 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0839bfbb_a478_424d_97e3_1438ba3a6a45.slice/crio-91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5 WatchSource:0}: Error finding container 91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5: Status 404 returned error can't find the container with id 91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5 Sep 10 13:45:50.218177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:50.218084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" event={"ID":"0839bfbb-a478-424d-97e3-1438ba3a6a45","Type":"ContainerStarted","Data":"13dba4b8ba63d3fa487147ee95201f12fb51868db9d51e9a2d61f66b39518974"} Sep 10 13:45:50.218177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:50.218168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" event={"ID":"0839bfbb-a478-424d-97e3-1438ba3a6a45","Type":"ContainerStarted","Data":"91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5"} Sep 10 13:45:50.250573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:50.250506 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" podStartSLOduration=2.250485607 podStartE2EDuration="2.250485607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:50.247839225 +0000 UTC m=+925.857624141" watchObservedRunningTime="2026-09-10 13:45:50.250485607 +0000 UTC m=+925.860270521" Sep 10 13:45:50.971208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:50.971119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:51.227188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:51.227097 2559 generic.go:356] "Generic (PLEG): container finished" podID="0839bfbb-a478-424d-97e3-1438ba3a6a45" containerID="13dba4b8ba63d3fa487147ee95201f12fb51868db9d51e9a2d61f66b39518974" exitCode=0 Sep 10 13:45:51.227691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:51.227188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" event={"ID":"0839bfbb-a478-424d-97e3-1438ba3a6a45","Type":"ContainerDied","Data":"13dba4b8ba63d3fa487147ee95201f12fb51868db9d51e9a2d61f66b39518974"} Sep 10 13:45:52.396030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:52.396001 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:52.494137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:52.494063 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca\") pod \"0839bfbb-a478-424d-97e3-1438ba3a6a45\" (UID: \"0839bfbb-a478-424d-97e3-1438ba3a6a45\") " Sep 10 13:45:52.494459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:52.494432 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0839bfbb-a478-424d-97e3-1438ba3a6a45" (UID: "0839bfbb-a478-424d-97e3-1438ba3a6a45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:52.595226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:52.595199 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0839bfbb-a478-424d-97e3-1438ba3a6a45-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:45:52.810245 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:52.810214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:53.249310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:53.249231 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" Sep 10 13:45:53.249310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:53.249262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn" event={"ID":"0839bfbb-a478-424d-97e3-1438ba3a6a45","Type":"ContainerDied","Data":"91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5"} Sep 10 13:45:53.249310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:53.249311 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a688d1aa99da784f666502a81eec23bb61626f1d9f052adfceffaf3ca1ddd5" Sep 10 13:45:54.430907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:54.430870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:54.596553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:54.596515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:56.082436 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.082399 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:56.273338 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.273308 2559 generic.go:356] "Generic (PLEG): container finished" podID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerID="b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde" exitCode=0 Sep 10 13:45:56.273497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.273359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerDied","Data":"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde"} Sep 10 13:45:56.273762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.273744 2559 scope.go:117] "RemoveContainer" containerID="b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde" Sep 10 13:45:56.274819 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.274790 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:56.274887 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.274865 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:56.274930 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.274902 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:45:56.491063 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.491026 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw"] Sep 10 13:45:56.491864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.491837 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0839bfbb-a478-424d-97e3-1438ba3a6a45" containerName="revoke-keys" Sep 10 13:45:56.492044 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.492029 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0839bfbb-a478-424d-97e3-1438ba3a6a45" containerName="revoke-keys" Sep 10 13:45:56.492318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.492300 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0839bfbb-a478-424d-97e3-1438ba3a6a45" containerName="revoke-keys" Sep 10 13:45:56.496259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.496228 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:56.508611 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.508591 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:45:56.521319 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.521295 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw"] Sep 10 13:45:56.633292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.633206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw\" (UID: \"17985a51-bfdf-42e8-984a-7fbb85ead3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:56.734009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.733961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw\" (UID: \"17985a51-bfdf-42e8-984a-7fbb85ead3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:56.734908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.734878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw\" (UID: \"17985a51-bfdf-42e8-984a-7fbb85ead3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:56.809974 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:56.809921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:57.123566 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.123536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw"] Sep 10 13:45:57.124183 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:45:57.124160 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17985a51_bfdf_42e8_984a_7fbb85ead3c2.slice/crio-b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52 WatchSource:0}: Error finding container b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52: Status 404 returned error can't find the container with id b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52 Sep 10 13:45:57.282440 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.282404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerStarted","Data":"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69"} Sep 10 13:45:57.283843 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.283814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" event={"ID":"17985a51-bfdf-42e8-984a-7fbb85ead3c2","Type":"ContainerStarted","Data":"b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52"} Sep 10 13:45:57.647723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.647622 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:45:57.648525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.648054 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:45:57.648525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.648116 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:45:57.704708 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:57.704683 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:58.292523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:58.292485 2559 generic.go:356] "Generic (PLEG): container finished" podID="17985a51-bfdf-42e8-984a-7fbb85ead3c2" containerID="73a98f083dccc6ce8ae5d587a7d9d5e02a66020218a407cc54b2223afdfe8ce8" exitCode=0 Sep 10 13:45:58.292885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:58.292579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" event={"ID":"17985a51-bfdf-42e8-984a-7fbb85ead3c2","Type":"ContainerDied","Data":"73a98f083dccc6ce8ae5d587a7d9d5e02a66020218a407cc54b2223afdfe8ce8"} Sep 10 13:45:58.646779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:58.646733 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:45:58.646985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:58.646804 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:45:59.014087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.014051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:45:59.479664 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.479641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:45:59.575355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.574825 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca\") pod \"17985a51-bfdf-42e8-984a-7fbb85ead3c2\" (UID: \"17985a51-bfdf-42e8-984a-7fbb85ead3c2\") " Sep 10 13:45:59.575732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.575691 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17985a51-bfdf-42e8-984a-7fbb85ead3c2" (UID: "17985a51-bfdf-42e8-984a-7fbb85ead3c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:45:59.649715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.649680 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:45:59.649864 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.649742 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:45:59.675774 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:45:59.675754 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17985a51-bfdf-42e8-984a-7fbb85ead3c2-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:00.306560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.306512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" event={"ID":"17985a51-bfdf-42e8-984a-7fbb85ead3c2","Type":"ContainerDied","Data":"b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52"} Sep 10 13:46:00.306769 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.306645 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81114ed858d646221504f76ad4427b763270e6d2572c66e94a1bb1765923e52" Sep 10 13:46:00.306769 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.306582 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw" Sep 10 13:46:00.386037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.386005 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:00.647281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.646813 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:46:00.647281 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:00.646871 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:46:01.646779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:01.646743 2559 patch_prober.go:28] interesting pod/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 10 13:46:01.647040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:01.647012 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 10 13:46:01.931579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:01.931494 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:46:01.982802 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:01.982770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:02.091030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:02.090999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:46:02.649910 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:02.649882 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:46:02.650905 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:02.650886 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:46:03.334201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:03.334155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:04.609390 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.609333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:04.615011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.614985 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n"] Sep 10 13:46:04.619607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.619584 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-ba18dc-29817465-jvk4n"] Sep 10 13:46:04.666421 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.666395 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:46:04.666693 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.666668 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" podUID="8955ce78-51f8-44fe-9582-ce2de64dccca" containerName="payload-processing" containerID="cri-o://9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064" gracePeriod=30 Sep 10 13:46:04.689942 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.689910 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:46:04.690164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.690134 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" podUID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" containerName="payload-pre-processing" containerID="cri-o://1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25" gracePeriod=30 Sep 10 13:46:04.750435 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.750406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:04.921505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.921468 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e1dcc5-1a75-435f-a18e-1fae156c13ef" path="/var/lib/kubelet/pods/d0e1dcc5-1a75-435f-a18e-1fae156c13ef/volumes" Sep 10 13:46:04.992686 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.992661 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:46:04.996420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:04.996397 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:46:05.083931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.083895 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:05.131012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.130985 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kghmg\" (UniqueName: \"kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg\") pod \"8955ce78-51f8-44fe-9582-ce2de64dccca\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " Sep 10 13:46:05.131157 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131080 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume\") pod \"8955ce78-51f8-44fe-9582-ce2de64dccca\" (UID: \"8955ce78-51f8-44fe-9582-ce2de64dccca\") " Sep 10 13:46:05.131246 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131227 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvpdl\" (UniqueName: \"kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl\") pod \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " Sep 10 13:46:05.131285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131276 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume\") pod \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\" (UID: \"00f26c61-45dc-422e-8cfe-4d3c27a2caa5\") " Sep 10 13:46:05.131446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131422 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8955ce78-51f8-44fe-9582-ce2de64dccca" (UID: "8955ce78-51f8-44fe-9582-ce2de64dccca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:05.131700 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131675 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00f26c61-45dc-422e-8cfe-4d3c27a2caa5" (UID: "00f26c61-45dc-422e-8cfe-4d3c27a2caa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:05.131789 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131771 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8955ce78-51f8-44fe-9582-ce2de64dccca-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.131857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.131795 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.133192 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.133166 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg" (OuterVolumeSpecName: "kube-api-access-kghmg") pod "8955ce78-51f8-44fe-9582-ce2de64dccca" (UID: "8955ce78-51f8-44fe-9582-ce2de64dccca"). InnerVolumeSpecName "kube-api-access-kghmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:05.133299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.133211 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl" (OuterVolumeSpecName: "kube-api-access-bvpdl") pod "00f26c61-45dc-422e-8cfe-4d3c27a2caa5" (UID: "00f26c61-45dc-422e-8cfe-4d3c27a2caa5"). InnerVolumeSpecName "kube-api-access-bvpdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:05.160267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.160191 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:46:05.160478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.160448 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" podUID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" containerName="payload-processing" containerID="cri-o://7d08c1bfc4028146ed738262a6c076c1df205e04a2d5894663929280b7dc99af" gracePeriod=30 Sep 10 13:46:05.198957 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.198925 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:46:05.199179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.199153 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" podUID="8714dcb2-e146-4507-82e5-64e073f4b11e" containerName="payload-pre-processing" containerID="cri-o://582a882478c8ef87fde1cd1c5867b20922714031c1ece99e7ed54d40112a0c12" gracePeriod=30 Sep 10 13:46:05.232950 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.232925 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghmg\" (UniqueName: \"kubernetes.io/projected/8955ce78-51f8-44fe-9582-ce2de64dccca-kube-api-access-kghmg\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.232950 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.232949 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvpdl\" (UniqueName: \"kubernetes.io/projected/00f26c61-45dc-422e-8cfe-4d3c27a2caa5-kube-api-access-bvpdl\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.309898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.309866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:05.346522 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.346482 2559 generic.go:356] "Generic (PLEG): container finished" podID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" containerID="1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25" exitCode=0 Sep 10 13:46:05.346522 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.346523 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" Sep 10 13:46:05.346782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.346634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" event={"ID":"00f26c61-45dc-422e-8cfe-4d3c27a2caa5","Type":"ContainerDied","Data":"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25"} Sep 10 13:46:05.346782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.346694 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt" event={"ID":"00f26c61-45dc-422e-8cfe-4d3c27a2caa5","Type":"ContainerDied","Data":"6e0cf91450356773965e37c3009d6bb6221292e769fbe795432e932cbbec745e"} Sep 10 13:46:05.346782 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.346726 2559 scope.go:117] "RemoveContainer" containerID="1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25" Sep 10 13:46:05.348712 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.348683 2559 generic.go:356] "Generic (PLEG): container finished" podID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" containerID="7d08c1bfc4028146ed738262a6c076c1df205e04a2d5894663929280b7dc99af" exitCode=0 Sep 10 13:46:05.348853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.348770 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" event={"ID":"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac","Type":"ContainerDied","Data":"7d08c1bfc4028146ed738262a6c076c1df205e04a2d5894663929280b7dc99af"} Sep 10 13:46:05.350423 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.350397 2559 generic.go:356] "Generic (PLEG): container finished" podID="8955ce78-51f8-44fe-9582-ce2de64dccca" containerID="9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064" exitCode=0 Sep 10 13:46:05.350561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.350440 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" Sep 10 13:46:05.350561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.350496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" event={"ID":"8955ce78-51f8-44fe-9582-ce2de64dccca","Type":"ContainerDied","Data":"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064"} Sep 10 13:46:05.350561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.350537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw" event={"ID":"8955ce78-51f8-44fe-9582-ce2de64dccca","Type":"ContainerDied","Data":"71aaa6ad9d8a124ddbda0e18e18534559ca320f7346c15838e6f7c287e8ec55a"} Sep 10 13:46:05.353454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.353159 2559 generic.go:356] "Generic (PLEG): container finished" podID="8714dcb2-e146-4507-82e5-64e073f4b11e" containerID="582a882478c8ef87fde1cd1c5867b20922714031c1ece99e7ed54d40112a0c12" exitCode=0 Sep 10 13:46:05.353454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.353208 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" event={"ID":"8714dcb2-e146-4507-82e5-64e073f4b11e","Type":"ContainerDied","Data":"582a882478c8ef87fde1cd1c5867b20922714031c1ece99e7ed54d40112a0c12"} Sep 10 13:46:05.359247 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.359228 2559 scope.go:117] "RemoveContainer" containerID="1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25" Sep 10 13:46:05.359619 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:46:05.359593 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25\": container with ID starting with 1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25 not found: ID does not exist" containerID="1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25" Sep 10 13:46:05.359690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.359630 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25"} err="failed to get container status \"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25\": rpc error: code = NotFound desc = could not find container \"1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25\": container with ID starting with 1c5aa042e079dd70a66b22061a2b5c02c156374ad70af364b32f8aeee22a5d25 not found: ID does not exist" Sep 10 13:46:05.359690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.359652 2559 scope.go:117] "RemoveContainer" containerID="9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064" Sep 10 13:46:05.377430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.377404 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn"] Sep 10 13:46:05.377518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.377449 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ba18dc-tvxjn"] Sep 10 13:46:05.377596 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.377559 2559 scope.go:117] "RemoveContainer" containerID="9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064" Sep 10 13:46:05.377923 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:46:05.377898 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064\": container with ID starting with 9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064 not found: ID does not exist" containerID="9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064" Sep 10 13:46:05.377993 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.377941 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064"} err="failed to get container status \"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064\": rpc error: code = NotFound desc = could not find container \"9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064\": container with ID starting with 9ef5dd660bf457c671dc63a49834c90fd14d89b9826f0ca5ed8772cd42ade064 not found: ID does not exist" Sep 10 13:46:05.431232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.431163 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:46:05.449785 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.449754 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ba18dc-fc9fbc8f8-7s7tt"] Sep 10 13:46:05.469511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.469488 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:46:05.472264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.472243 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:46:05.497334 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.497305 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:46:05.518848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.518810 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ba18dc-7874cf5cf9-pwmfw"] Sep 10 13:46:05.578752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.578716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:05.636875 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.636837 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume\") pod \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " Sep 10 13:46:05.637343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637002 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h9g7\" (UniqueName: \"kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7\") pod \"8714dcb2-e146-4507-82e5-64e073f4b11e\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " Sep 10 13:46:05.637343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637058 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume\") pod \"8714dcb2-e146-4507-82e5-64e073f4b11e\" (UID: \"8714dcb2-e146-4507-82e5-64e073f4b11e\") " Sep 10 13:46:05.637343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7m9j\" (UniqueName: \"kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j\") pod \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\" (UID: \"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac\") " Sep 10 13:46:05.637343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637325 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" (UID: "d15385cf-3d88-4a9b-bacb-e4e0c981b7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:05.637601 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637443 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.637831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.637796 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8714dcb2-e146-4507-82e5-64e073f4b11e" (UID: "8714dcb2-e146-4507-82e5-64e073f4b11e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:05.639595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.639566 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7" (OuterVolumeSpecName: "kube-api-access-4h9g7") pod "8714dcb2-e146-4507-82e5-64e073f4b11e" (UID: "8714dcb2-e146-4507-82e5-64e073f4b11e"). InnerVolumeSpecName "kube-api-access-4h9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:05.639967 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.639945 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j" (OuterVolumeSpecName: "kube-api-access-z7m9j") pod "d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" (UID: "d15385cf-3d88-4a9b-bacb-e4e0c981b7ac"). InnerVolumeSpecName "kube-api-access-z7m9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:05.738073 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.738037 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9g7\" (UniqueName: \"kubernetes.io/projected/8714dcb2-e146-4507-82e5-64e073f4b11e-kube-api-access-4h9g7\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.738073 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.738066 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8714dcb2-e146-4507-82e5-64e073f4b11e-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.738250 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.738085 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7m9j\" (UniqueName: \"kubernetes.io/projected/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac-kube-api-access-z7m9j\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:05.760850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.760814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:05.962649 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:05.962551 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.094032 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.094004 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw"] Sep 10 13:46:06.096945 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.096916 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:46:06.097162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.097137 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" containerID="cri-o://55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69" gracePeriod=30 Sep 10 13:46:06.099248 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.099227 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e404ac71-6lfqw"] Sep 10 13:46:06.134398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.134349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.294328 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.294284 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:06.360278 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.360248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" event={"ID":"d15385cf-3d88-4a9b-bacb-e4e0c981b7ac","Type":"ContainerDied","Data":"827815c31f26120a8c71e1e8f31698f098adbf21548d4d0af6cb6bbaeb36c75b"} Sep 10 13:46:06.360471 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.360257 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6" Sep 10 13:46:06.360471 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.360343 2559 scope.go:117] "RemoveContainer" containerID="7d08c1bfc4028146ed738262a6c076c1df205e04a2d5894663929280b7dc99af" Sep 10 13:46:06.362908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.362889 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" Sep 10 13:46:06.363023 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.362923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww" event={"ID":"8714dcb2-e146-4507-82e5-64e073f4b11e","Type":"ContainerDied","Data":"a0ed9f360dcfc1f5f044a7d9a8c297fb69897e4650193d9b20fd0e328d766b14"} Sep 10 13:46:06.370819 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.370799 2559 scope.go:117] "RemoveContainer" containerID="582a882478c8ef87fde1cd1c5867b20922714031c1ece99e7ed54d40112a0c12" Sep 10 13:46:06.425610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.425587 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:46:06.439526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.439506 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e404ac71-74d6d9cb5-jqxs6"] Sep 10 13:46:06.460096 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.460066 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:46:06.466604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.466586 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e404ac71-78ccb64454-7vxww"] Sep 10 13:46:06.918308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.918281 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" path="/var/lib/kubelet/pods/00f26c61-45dc-422e-8cfe-4d3c27a2caa5/volumes" Sep 10 13:46:06.918656 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.918618 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0839bfbb-a478-424d-97e3-1438ba3a6a45" path="/var/lib/kubelet/pods/0839bfbb-a478-424d-97e3-1438ba3a6a45/volumes" Sep 10 13:46:06.918907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.918894 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17985a51-bfdf-42e8-984a-7fbb85ead3c2" path="/var/lib/kubelet/pods/17985a51-bfdf-42e8-984a-7fbb85ead3c2/volumes" Sep 10 13:46:06.919238 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.919225 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8714dcb2-e146-4507-82e5-64e073f4b11e" path="/var/lib/kubelet/pods/8714dcb2-e146-4507-82e5-64e073f4b11e/volumes" Sep 10 13:46:06.919880 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.919865 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8955ce78-51f8-44fe-9582-ce2de64dccca" path="/var/lib/kubelet/pods/8955ce78-51f8-44fe-9582-ce2de64dccca/volumes" Sep 10 13:46:06.920185 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:06.920173 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" path="/var/lib/kubelet/pods/d15385cf-3d88-4a9b-bacb-e4e0c981b7ac/volumes" Sep 10 13:46:07.088031 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:07.088006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:07.232504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:07.232433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:08.078190 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:08.078155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.019432 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:09.019388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:09.129923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:09.129892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:10.520614 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:10.520579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:10.922711 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:10.922628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:11.345645 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.345622 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:46:11.384747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384716 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.384896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384760 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.384896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384799 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.384896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktqll\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.384896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384884 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384920 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.384994 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.385032 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.385040 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.385109 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.385069 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo\") pod \"19bf6fac-4469-4f51-99ea-64d57623fe0d\" (UID: \"19bf6fac-4469-4f51-99ea-64d57623fe0d\") " Sep 10 13:46:11.386337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.385815 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:46:11.386337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.386189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.387934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.386691 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.387934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.386755 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.387934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.386723 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.387934 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.386921 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bf6fac-4469-4f51-99ea-64d57623fe0d-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.397981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.397917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:11.398760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.398709 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:46:11.398889 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.398756 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll" (OuterVolumeSpecName: "kube-api-access-ktqll") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "kube-api-access-ktqll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:46:11.399147 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.399123 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19bf6fac-4469-4f51-99ea-64d57623fe0d" (UID: "19bf6fac-4469-4f51-99ea-64d57623fe0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:46:11.403585 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.403556 2559 generic.go:356] "Generic (PLEG): container finished" podID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerID="55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69" exitCode=0 Sep 10 13:46:11.403697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.403614 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" Sep 10 13:46:11.403697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.403632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerDied","Data":"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69"} Sep 10 13:46:11.403826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.403733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2" event={"ID":"19bf6fac-4469-4f51-99ea-64d57623fe0d","Type":"ContainerDied","Data":"f316152813d88542d10fe5de17ba0a72d8aa993028c03ff61e9c493573ee654d"} Sep 10 13:46:11.403826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.403782 2559 scope.go:117] "RemoveContainer" containerID="55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69" Sep 10 13:46:11.414584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.414566 2559 scope.go:117] "RemoveContainer" containerID="b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde" Sep 10 13:46:11.423172 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.423153 2559 scope.go:117] "RemoveContainer" containerID="55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69" Sep 10 13:46:11.423416 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:46:11.423400 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69\": container with ID starting with 55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69 not found: ID does not exist" containerID="55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69" Sep 10 13:46:11.423466 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.423423 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69"} err="failed to get container status \"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69\": rpc error: code = NotFound desc = could not find container \"55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69\": container with ID starting with 55972be390e1d2eaa6c139c6b9fae9c5b9aad3e14a36424d98af44f9c1f41e69 not found: ID does not exist" Sep 10 13:46:11.423466 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.423438 2559 scope.go:117] "RemoveContainer" containerID="b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde" Sep 10 13:46:11.423678 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:46:11.423663 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde\": container with ID starting with b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde not found: ID does not exist" containerID="b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde" Sep 10 13:46:11.423720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.423682 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde"} err="failed to get container status \"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde\": rpc error: code = NotFound desc = could not find container \"b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde\": container with ID starting with b471d791295155790cde5747f75809affb9bb50dcf69d0289e84ed8911e71dde not found: ID does not exist" Sep 10 13:46:11.457872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.457843 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:46:11.461115 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.461093 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e404ac71-openshift-default-54768bd444-xgfq2"] Sep 10 13:46:11.488099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488047 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.488099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488068 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqll\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-kube-api-access-ktqll\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.488099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488084 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.488099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488097 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.488277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488110 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:11.488277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:11.488122 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bf6fac-4469-4f51-99ea-64d57623fe0d-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:46:12.918525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:12.918493 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" path="/var/lib/kubelet/pods/19bf6fac-4469-4f51-99ea-64d57623fe0d/volumes" Sep 10 13:46:16.750654 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:16.750621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:16.859414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:16.859382 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:16.981127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:16.981095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:17.233006 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:17.232977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:17.701761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:17.701732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:21.931215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:21.931178 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:46:30.324087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:30.324051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:33.798435 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:33.798402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:34.499793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:34.499755 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:46:41.165088 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:41.165049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:41.564925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:41.564889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:42.070597 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.070556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:42.496027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.495949 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:46:42.496583 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496563 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17985a51-bfdf-42e8-984a-7fbb85ead3c2" containerName="revoke-keys" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496585 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="17985a51-bfdf-42e8-984a-7fbb85ead3c2" containerName="revoke-keys" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496609 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8955ce78-51f8-44fe-9582-ce2de64dccca" containerName="payload-processing" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496618 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8955ce78-51f8-44fe-9582-ce2de64dccca" containerName="payload-processing" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496633 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" containerName="payload-pre-processing" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496642 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" containerName="payload-pre-processing" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496655 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" containerName="payload-processing" Sep 10 13:46:42.496674 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496663 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" containerName="payload-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496690 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8714dcb2-e146-4507-82e5-64e073f4b11e" containerName="payload-pre-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496699 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8714dcb2-e146-4507-82e5-64e073f4b11e" containerName="payload-pre-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496712 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496722 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496732 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496740 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496859 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8955ce78-51f8-44fe-9582-ce2de64dccca" containerName="payload-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496872 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496883 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f26c61-45dc-422e-8cfe-4d3c27a2caa5" containerName="payload-pre-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496894 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8714dcb2-e146-4507-82e5-64e073f4b11e" containerName="payload-pre-processing" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496906 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bf6fac-4469-4f51-99ea-64d57623fe0d" containerName="istio-proxy" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496915 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="17985a51-bfdf-42e8-984a-7fbb85ead3c2" containerName="revoke-keys" Sep 10 13:46:42.497019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.496925 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15385cf-3d88-4a9b-bacb-e4e0c981b7ac" containerName="payload-processing" Sep 10 13:46:42.501895 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.501872 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.507525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.507500 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7eb0a3-openshift-default-dockercfg-klrpq\"" Sep 10 13:46:42.532397 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.532355 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:46:42.542386 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.542342 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:42.645446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86sn\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.645803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.645749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.688375 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.688337 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:42.746779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.746706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.746779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.746748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.746932 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.746887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.746987 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.746947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86sn\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.746987 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.746977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747333 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747333 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747549 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747817 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747817 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.747966 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.747934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.749176 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.749154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.749553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.749534 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.766923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.766900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.767389 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.767352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86sn\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn\") pod \"e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:42.816523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:42.816499 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:43.111480 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.111453 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:46:43.113149 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:46:43.113123 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a47e330_0507_4521_b593_9c7c69da6341.slice/crio-764c92e2ef878abc12fbce4c29e1d92ca1fa8563fb67ba5db55dc021c8472122 WatchSource:0}: Error finding container 764c92e2ef878abc12fbce4c29e1d92ca1fa8563fb67ba5db55dc021c8472122: Status 404 returned error can't find the container with id 764c92e2ef878abc12fbce4c29e1d92ca1fa8563fb67ba5db55dc021c8472122 Sep 10 13:46:43.115285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.115253 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:46:43.115378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.115313 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:46:43.115378 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.115341 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:46:43.613789 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.613756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerStarted","Data":"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23"} Sep 10 13:46:43.614246 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.613806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerStarted","Data":"764c92e2ef878abc12fbce4c29e1d92ca1fa8563fb67ba5db55dc021c8472122"} Sep 10 13:46:43.656459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.656408 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podStartSLOduration=1.65639279 podStartE2EDuration="1.65639279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:43.652397703 +0000 UTC m=+979.262182615" watchObservedRunningTime="2026-09-10 13:46:43.65639279 +0000 UTC m=+979.266177700" Sep 10 13:46:43.753533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.753504 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:43.817492 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.817457 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:43.909441 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:43.909359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:44.025232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:44.025204 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:44.682500 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:44.682466 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:44.822543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:44.822517 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:45.115263 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:45.115230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:45.628346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:45.628319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:45.629194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:45.629170 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:46:45.799676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:45.799641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:45.895889 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:45.895820 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:46.335114 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:46.335061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:47.467279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:47.467178 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:48.550669 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:48.550633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:48.713599 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:48.713561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:48.982540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:48.982440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:49.416588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.416557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:49.518541 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.518511 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:46:49.522310 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.522294 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.525209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.525177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7eb0a3\"" Sep 10 13:46:49.525343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.525220 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7eb0a3-dockercfg-vhrt2\"" Sep 10 13:46:49.534778 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.534756 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:46:49.554799 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.554774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:49.612449 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.612408 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.612849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.612825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2h4\" (UniqueName: \"kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.714352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.714276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2h4\" (UniqueName: \"kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.714352 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.714320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.715041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.715021 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.724190 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.724169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2h4\" (UniqueName: \"kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4\") pod \"payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:49.833259 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:49.833232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:50.120527 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:50.120499 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:46:50.121674 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:46:50.121653 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3865aa15_1863_4d2a_955e_031a73369bbc.slice/crio-0f9216701384a069a3999bad0a6da387dfcf520061d9b41a583410d6c55ad3c6 WatchSource:0}: Error finding container 0f9216701384a069a3999bad0a6da387dfcf520061d9b41a583410d6c55ad3c6: Status 404 returned error can't find the container with id 0f9216701384a069a3999bad0a6da387dfcf520061d9b41a583410d6c55ad3c6 Sep 10 13:46:50.662848 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:50.662814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" event={"ID":"3865aa15-1863-4d2a-955e-031a73369bbc","Type":"ContainerStarted","Data":"e0f010568193aaabb166f8471bb793db8ea56f37ce9b5285c986f7e57065b6ce"} Sep 10 13:46:50.663339 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:50.662865 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" event={"ID":"3865aa15-1863-4d2a-955e-031a73369bbc","Type":"ContainerStarted","Data":"0f9216701384a069a3999bad0a6da387dfcf520061d9b41a583410d6c55ad3c6"} Sep 10 13:46:50.663339 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:50.662915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:46:50.707103 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:50.707060 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" podStartSLOduration=1.707048436 podStartE2EDuration="1.707048436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:50.702501461 +0000 UTC m=+986.312286373" watchObservedRunningTime="2026-09-10 13:46:50.707048436 +0000 UTC m=+986.316833348" Sep 10 13:46:52.382492 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:52.382457 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:52.492540 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:52.492505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:55.058514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:55.058477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:46:56.339560 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:56.339525 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:46:56.682311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:46:56.682238 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:00.801162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:00.801126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:00.901831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:00.901795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:01.670394 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:01.670349 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:47:06.786570 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.786540 2559 generic.go:356] "Generic (PLEG): container finished" podID="3a47e330-0507-4521-b593-9c7c69da6341" containerID="7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23" exitCode=0 Sep 10 13:47:06.786968 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.786609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerDied","Data":"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23"} Sep 10 13:47:06.787041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.787002 2559 scope.go:117] "RemoveContainer" containerID="7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23" Sep 10 13:47:06.788051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.788022 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:06.788129 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.788110 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:06.788168 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:06.788159 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:07.608037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:07.607999 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:07.794768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:07.794737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerStarted","Data":"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9"} Sep 10 13:47:07.817153 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:07.817129 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:47:07.817323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:07.817303 2559 patch_prober.go:28] interesting pod/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:47:07.817390 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:07.817348 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 13:47:08.817582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:08.817553 2559 patch_prober.go:28] interesting pod/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:47:08.817974 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:08.817601 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 13:47:09.816877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:09.816835 2559 patch_prober.go:28] interesting pod/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:47:09.817090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:09.816904 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 13:47:10.817486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:10.817448 2559 patch_prober.go:28] interesting pod/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:47:10.817911 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:10.817507 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 13:47:11.045414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:11.045354 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:47:11.817854 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:11.817816 2559 patch_prober.go:28] interesting pod/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Sep 10 13:47:11.818267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:11.817896 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Sep 10 13:47:12.855476 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:12.855444 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:47:12.855940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:12.855869 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:47:12.856533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:12.856514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:47:13.134891 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:13.134808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:14.131028 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:14.130995 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:14.978298 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:14.978264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:15.350691 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:15.350658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:15.461983 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:15.461945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:15.619661 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:15.619570 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:15.720722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:15.720688 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:15.971887 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:15.971795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:16.329360 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:16.329312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:26.215082 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:26.215047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:27.106915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:27.106871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:27.502231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:27.502142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:27.803380 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:27.803344 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:47:34.997623 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:34.997590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:35.117571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:35.117526 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:35.307700 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:35.307667 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:35.463316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:35.463280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:36.124064 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:36.124025 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:36.215779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:36.215745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:36.428853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:36.428775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:38.037244 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:38.037210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:41.534460 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:41.534423 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:47:41.588790 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:41.588746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.180087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:45.180050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.281928 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:45.281892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:45.927037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:45.927008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:53.092437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.092403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:53.203964 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.203930 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:53.463290 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.463211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:53.720355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.720266 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:47:53.724048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.724026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.726998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.726977 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-21316764-openshift-default-dockercfg-bzxmf\"" Sep 10 13:47:53.744008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.743987 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:47:53.785993 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.785967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:53.893533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893732 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82p5q\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.893916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.893870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82p5q\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.994834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.994837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.995016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.995093 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.995140 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.995169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.995591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.995571 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.997051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.997028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:53.997274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:53.997259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:54.005126 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.005098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:54.005323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.005302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82p5q\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q\") pod \"e2e-mt-21316764-openshift-default-fb968c688-gfrtn\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:54.008186 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.008155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:54.034775 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.034758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:54.325901 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:47:54.325852 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bcfb10_5224_4481_a870_2828ae52aff2.slice/crio-4626dba282e12cd1871ef331ac5063599d96d86e2c238862aae19aca1a565b82 WatchSource:0}: Error finding container 4626dba282e12cd1871ef331ac5063599d96d86e2c238862aae19aca1a565b82: Status 404 returned error can't find the container with id 4626dba282e12cd1871ef331ac5063599d96d86e2c238862aae19aca1a565b82 Sep 10 13:47:54.326584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.326556 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:47:54.328178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.328148 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:54.328264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.328208 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:54.328264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.328235 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:47:54.919173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:54.919138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:55.031461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.031432 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g"] Sep 10 13:47:55.035351 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.035335 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:55.038301 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.038280 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:47:55.045591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.045569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g"] Sep 10 13:47:55.093218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.093190 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerStarted","Data":"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae"} Sep 10 13:47:55.093342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.093243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerStarted","Data":"4626dba282e12cd1871ef331ac5063599d96d86e2c238862aae19aca1a565b82"} Sep 10 13:47:55.125876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.125831 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podStartSLOduration=2.125815734 podStartE2EDuration="2.125815734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:55.123859559 +0000 UTC m=+1050.733644472" watchObservedRunningTime="2026-09-10 13:47:55.125815734 +0000 UTC m=+1050.735600647" Sep 10 13:47:55.207078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.207020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g\" (UID: \"80477e58-32ef-4549-bbf6-7d2ef6412010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:55.308078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.308040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g\" (UID: \"80477e58-32ef-4549-bbf6-7d2ef6412010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:55.308828 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.308807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g\" (UID: \"80477e58-32ef-4549-bbf6-7d2ef6412010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:55.346888 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.346862 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:55.638719 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:55.638689 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g"] Sep 10 13:47:55.639036 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:47:55.639015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80477e58_32ef_4549_bbf6_7d2ef6412010.slice/crio-12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca WatchSource:0}: Error finding container 12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca: Status 404 returned error can't find the container with id 12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca Sep 10 13:47:56.035948 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.035920 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:56.040867 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.040845 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:56.105892 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.105855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" event={"ID":"80477e58-32ef-4549-bbf6-7d2ef6412010","Type":"ContainerStarted","Data":"88a8a6a8fa11e240c3fbc9d6557a43dc5f437e62c29a2efe99c90bc0d59f2411"} Sep 10 13:47:56.106040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.105908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" event={"ID":"80477e58-32ef-4549-bbf6-7d2ef6412010","Type":"ContainerStarted","Data":"12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca"} Sep 10 13:47:56.106107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.106047 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:56.107011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.106995 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:47:56.134866 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.134821 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" podStartSLOduration=1.134809672 podStartE2EDuration="1.134809672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:56.132854021 +0000 UTC m=+1051.742638937" watchObservedRunningTime="2026-09-10 13:47:56.134809672 +0000 UTC m=+1051.744594585" Sep 10 13:47:56.216203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:56.214011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:57.114065 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:57.114028 2559 generic.go:356] "Generic (PLEG): container finished" podID="80477e58-32ef-4549-bbf6-7d2ef6412010" containerID="88a8a6a8fa11e240c3fbc9d6557a43dc5f437e62c29a2efe99c90bc0d59f2411" exitCode=0 Sep 10 13:47:57.114452 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:57.114084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" event={"ID":"80477e58-32ef-4549-bbf6-7d2ef6412010","Type":"ContainerDied","Data":"88a8a6a8fa11e240c3fbc9d6557a43dc5f437e62c29a2efe99c90bc0d59f2411"} Sep 10 13:47:57.331183 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:57.331146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:57.478007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:57.477938 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:58.262792 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:58.262761 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:58.306159 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:58.306112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:47:58.437950 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:58.437880 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca\") pod \"80477e58-32ef-4549-bbf6-7d2ef6412010\" (UID: \"80477e58-32ef-4549-bbf6-7d2ef6412010\") " Sep 10 13:47:58.438236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:58.438212 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80477e58-32ef-4549-bbf6-7d2ef6412010" (UID: "80477e58-32ef-4549-bbf6-7d2ef6412010"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:47:58.538623 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:58.538598 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80477e58-32ef-4549-bbf6-7d2ef6412010-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:47:59.127680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:59.127653 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" Sep 10 13:47:59.127680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:59.127669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g" event={"ID":"80477e58-32ef-4549-bbf6-7d2ef6412010","Type":"ContainerDied","Data":"12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca"} Sep 10 13:47:59.127841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:59.127744 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a9f3e888d928b709932edbcc9f0053528c815ab33e6bc01a2bfb06b256d9ca" Sep 10 13:47:59.928237 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:47:59.928184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:00.653298 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:00.653262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:48:01.471989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:01.471950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:02.926880 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:02.926851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:03.072031 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:03.071993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:03.946555 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:03.946521 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:04.085695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.085661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:04.218400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.218297 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:04.218743 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.218731 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80477e58-32ef-4549-bbf6-7d2ef6412010" containerName="revoke-keys" Sep 10 13:48:04.218788 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.218745 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80477e58-32ef-4549-bbf6-7d2ef6412010" containerName="revoke-keys" Sep 10 13:48:04.218831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.218823 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80477e58-32ef-4549-bbf6-7d2ef6412010" containerName="revoke-keys" Sep 10 13:48:04.222235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.222217 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.232305 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.232284 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-21316764-dockercfg-4qwjl\"" Sep 10 13:48:04.233108 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.233092 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-21316764\"" Sep 10 13:48:04.248463 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.248434 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:04.313774 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.313747 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:04.317187 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.317171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.345832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.345811 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:04.389703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.389676 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.389821 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.389801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgqq\" (UniqueName: \"kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.490676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.490607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgqq\" (UniqueName: \"kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.490676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.490660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.490841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.490703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.490841 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.490766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74trt\" (UniqueName: \"kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.491425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.491408 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.504922 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.504898 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgqq\" (UniqueName: \"kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq\") pod \"payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.532739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.532712 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:04.592699 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.592356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.592699 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.592460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74trt\" (UniqueName: \"kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.593330 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.593288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.612668 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.612637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74trt\" (UniqueName: \"kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt\") pod \"payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.627341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.627317 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:04.831431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.831408 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:04.833352 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:48:04.833320 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231c056c_3ca0_4b84_9750_03b89af3bb4b.slice/crio-fda7766f71591cdbcadef14555843dda24002b686a34ea412973047803779f8a WatchSource:0}: Error finding container fda7766f71591cdbcadef14555843dda24002b686a34ea412973047803779f8a: Status 404 returned error can't find the container with id fda7766f71591cdbcadef14555843dda24002b686a34ea412973047803779f8a Sep 10 13:48:04.877045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.876987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:04.911018 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:48:04.910997 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635226e5_7ead_40e5_b628_7f8d90b8b659.slice/crio-ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea WatchSource:0}: Error finding container ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea: Status 404 returned error can't find the container with id ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea Sep 10 13:48:04.911723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:04.911699 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:05.173270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.173194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" event={"ID":"635226e5-7ead-40e5-b628-7f8d90b8b659","Type":"ContainerStarted","Data":"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6"} Sep 10 13:48:05.173270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.173244 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" event={"ID":"635226e5-7ead-40e5-b628-7f8d90b8b659","Type":"ContainerStarted","Data":"ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea"} Sep 10 13:48:05.173771 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.173354 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:05.175061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.175035 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" event={"ID":"231c056c-3ca0-4b84-9750-03b89af3bb4b","Type":"ContainerStarted","Data":"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75"} Sep 10 13:48:05.175188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.175085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" event={"ID":"231c056c-3ca0-4b84-9750-03b89af3bb4b","Type":"ContainerStarted","Data":"fda7766f71591cdbcadef14555843dda24002b686a34ea412973047803779f8a"} Sep 10 13:48:05.175188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.175116 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:05.226044 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.225991 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" podStartSLOduration=1.225975565 podStartE2EDuration="1.225975565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:05.222797671 +0000 UTC m=+1060.832582610" watchObservedRunningTime="2026-09-10 13:48:05.225975565 +0000 UTC m=+1060.835760523" Sep 10 13:48:05.269965 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:05.269923 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" podStartSLOduration=1.2699111379999999 podStartE2EDuration="1.269911138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:05.267307542 +0000 UTC m=+1060.877092445" watchObservedRunningTime="2026-09-10 13:48:05.269911138 +0000 UTC m=+1060.879696051" Sep 10 13:48:06.008314 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:06.008279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:06.897134 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:06.897082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:07.018965 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.018935 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:48:07.019199 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.019171 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" podUID="3865aa15-1863-4d2a-955e-031a73369bbc" containerName="payload-pre-processing" containerID="cri-o://e0f010568193aaabb166f8471bb793db8ea56f37ce9b5285c986f7e57065b6ce" gracePeriod=30 Sep 10 13:48:07.088975 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.088949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:07.191558 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.191525 2559 generic.go:356] "Generic (PLEG): container finished" podID="3865aa15-1863-4d2a-955e-031a73369bbc" containerID="e0f010568193aaabb166f8471bb793db8ea56f37ce9b5285c986f7e57065b6ce" exitCode=0 Sep 10 13:48:07.191689 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.191563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" event={"ID":"3865aa15-1863-4d2a-955e-031a73369bbc","Type":"ContainerDied","Data":"e0f010568193aaabb166f8471bb793db8ea56f37ce9b5285c986f7e57065b6ce"} Sep 10 13:48:07.275800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.275774 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g"] Sep 10 13:48:07.277552 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.277533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:48:07.281722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.281700 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7eb0a3-4fg9g"] Sep 10 13:48:07.315715 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.315691 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh2h4\" (UniqueName: \"kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4\") pod \"3865aa15-1863-4d2a-955e-031a73369bbc\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " Sep 10 13:48:07.315844 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.315807 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume\") pod \"3865aa15-1863-4d2a-955e-031a73369bbc\" (UID: \"3865aa15-1863-4d2a-955e-031a73369bbc\") " Sep 10 13:48:07.316164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.316143 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3865aa15-1863-4d2a-955e-031a73369bbc" (UID: "3865aa15-1863-4d2a-955e-031a73369bbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:07.317727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.317705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4" (OuterVolumeSpecName: "kube-api-access-qh2h4") pod "3865aa15-1863-4d2a-955e-031a73369bbc" (UID: "3865aa15-1863-4d2a-955e-031a73369bbc"). InnerVolumeSpecName "kube-api-access-qh2h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:07.321074 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.321050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:07.416733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.416703 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3865aa15-1863-4d2a-955e-031a73369bbc-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:07.416733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.416728 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2h4\" (UniqueName: \"kubernetes.io/projected/3865aa15-1863-4d2a-955e-031a73369bbc-kube-api-access-qh2h4\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:07.435759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.435681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:07.661205 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.661172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:07.824268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:07.824239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:08.199643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.199565 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" Sep 10 13:48:08.199643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.199590 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh" event={"ID":"3865aa15-1863-4d2a-955e-031a73369bbc","Type":"ContainerDied","Data":"0f9216701384a069a3999bad0a6da387dfcf520061d9b41a583410d6c55ad3c6"} Sep 10 13:48:08.200087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.199650 2559 scope.go:117] "RemoveContainer" containerID="e0f010568193aaabb166f8471bb793db8ea56f37ce9b5285c986f7e57065b6ce" Sep 10 13:48:08.230655 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.230630 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:48:08.236006 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.235983 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7eb0a3-5cbd44d4b-2w6nh"] Sep 10 13:48:08.813161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.813128 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:08.919315 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.919280 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3865aa15-1863-4d2a-955e-031a73369bbc" path="/var/lib/kubelet/pods/3865aa15-1863-4d2a-955e-031a73369bbc/volumes" Sep 10 13:48:08.919676 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:08.919662 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80477e58-32ef-4549-bbf6-7d2ef6412010" path="/var/lib/kubelet/pods/80477e58-32ef-4549-bbf6-7d2ef6412010/volumes" Sep 10 13:48:09.879768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:09.879731 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:10.779177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:10.779140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:13.443047 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:13.443014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:15.437053 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:15.437018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:15.573306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:15.573274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:15.615534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:15.615504 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:48:15.615742 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:15.615719 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" containerID="cri-o://403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9" gracePeriod=30 Sep 10 13:48:15.784141 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:15.784112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:16.100613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:16.100530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:16.184178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:16.184150 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:16.184340 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:16.184227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:16.463603 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:16.463506 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:17.252743 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:17.252645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:18.091970 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:18.091924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:18.220602 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:18.220558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:18.329970 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:18.329931 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:20.787546 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.787515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:48:20.865413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.865390 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:48:20.944576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944539 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944710 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944600 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944710 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944625 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n86sn\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944710 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944644 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944710 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944693 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944724 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944748 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944767 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy\") pod \"3a47e330-0507-4521-b593-9c7c69da6341\" (UID: \"3a47e330-0507-4521-b593-9c7c69da6341\") " Sep 10 13:48:20.944925 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944820 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:20.945169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.944974 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:20.945169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945043 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:20.945169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945047 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:20.945169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945143 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:20.945343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945281 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:20.945343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945301 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:20.945343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945319 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:20.945343 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.945333 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:20.947264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.947242 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:20.947264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.947256 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn" (OuterVolumeSpecName: "kube-api-access-n86sn") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "kube-api-access-n86sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:20.947463 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.947441 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:48:20.947508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:20.947482 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a47e330-0507-4521-b593-9c7c69da6341" (UID: "3a47e330-0507-4521-b593-9c7c69da6341"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:48:21.046169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.046147 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a47e330-0507-4521-b593-9c7c69da6341-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:21.046169 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.046167 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a47e330-0507-4521-b593-9c7c69da6341-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:21.046311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.046175 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a47e330-0507-4521-b593-9c7c69da6341-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:21.046311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.046183 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:21.046311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.046191 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86sn\" (UniqueName: \"kubernetes.io/projected/3a47e330-0507-4521-b593-9c7c69da6341-kube-api-access-n86sn\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:21.303762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.303685 2559 generic.go:356] "Generic (PLEG): container finished" podID="3a47e330-0507-4521-b593-9c7c69da6341" containerID="403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9" exitCode=0 Sep 10 13:48:21.303762 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.303732 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" Sep 10 13:48:21.303978 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.303768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerDied","Data":"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9"} Sep 10 13:48:21.303978 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.303825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm" event={"ID":"3a47e330-0507-4521-b593-9c7c69da6341","Type":"ContainerDied","Data":"764c92e2ef878abc12fbce4c29e1d92ca1fa8563fb67ba5db55dc021c8472122"} Sep 10 13:48:21.303978 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.303857 2559 scope.go:117] "RemoveContainer" containerID="403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9" Sep 10 13:48:21.305891 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.305856 2559 generic.go:356] "Generic (PLEG): container finished" podID="35bcfb10-5224-4481-a870-2828ae52aff2" containerID="96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae" exitCode=0 Sep 10 13:48:21.306021 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.305911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerDied","Data":"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae"} Sep 10 13:48:21.306440 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.306352 2559 scope.go:117] "RemoveContainer" containerID="96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae" Sep 10 13:48:21.307666 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.307632 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:48:21.307784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.307728 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:48:21.307784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.307777 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:48:21.316274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.316255 2559 scope.go:117] "RemoveContainer" containerID="7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23" Sep 10 13:48:21.348127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.347077 2559 scope.go:117] "RemoveContainer" containerID="403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9" Sep 10 13:48:21.349848 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:48:21.349696 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9\": container with ID starting with 403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9 not found: ID does not exist" containerID="403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9" Sep 10 13:48:21.349958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.349786 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9"} err="failed to get container status \"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9\": rpc error: code = NotFound desc = could not find container \"403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9\": container with ID starting with 403d7f10faf808f2b0cacfbf4cf02eaef0e16aaf51ddcca2485e16b4a4bd60e9 not found: ID does not exist" Sep 10 13:48:21.350036 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.349964 2559 scope.go:117] "RemoveContainer" containerID="7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23" Sep 10 13:48:21.383079 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:48:21.383032 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23\": container with ID starting with 7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23 not found: ID does not exist" containerID="7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23" Sep 10 13:48:21.383296 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.383267 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23"} err="failed to get container status \"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23\": rpc error: code = NotFound desc = could not find container \"7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23\": container with ID starting with 7be9a24d88917fb0869528ff949e6760063f17a0eb187f35af51121d026fdd23 not found: ID does not exist" Sep 10 13:48:21.397344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.397235 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:48:21.401862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:21.401614 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7eb0a3-openshift-default-5bdf4fdbd4-fwtbm"] Sep 10 13:48:22.314927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:22.314886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerStarted","Data":"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de"} Sep 10 13:48:22.918446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:22.918406 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a47e330-0507-4521-b593-9c7c69da6341" path="/var/lib/kubelet/pods/3a47e330-0507-4521-b593-9c7c69da6341/volumes" Sep 10 13:48:23.037309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:23.037268 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:48:23.037518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:23.037420 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 13:48:23.037591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:23.037506 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 13:48:24.035237 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:24.035198 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 13:48:24.035849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:24.035816 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 13:48:24.677503 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:24.677469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:25.036048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:25.036018 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 13:48:25.036532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:25.036500 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 13:48:26.035845 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:26.035802 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 13:48:26.036055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:26.035869 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 13:48:26.106218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:26.106186 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:48:26.633344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:26.633305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:27.035277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:27.035246 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 10 13:48:27.035577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:27.035523 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 10 13:48:27.319935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:27.319803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:27.621123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:27.620998 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.039137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.039109 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:48:28.039967 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.039949 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:48:28.054071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.054046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.169580 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.169546 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.531947 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.531918 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:28.893496 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:28.893410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:29.874137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:29.874103 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:35.107885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:35.107853 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:48:35.920739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:35.920692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:35.996462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:35.996440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:36.372966 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.372930 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7"] Sep 10 13:48:36.373393 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373362 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.373393 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373390 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.373486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373404 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.373486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373410 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.373486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373417 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3865aa15-1863-4d2a-955e-031a73369bbc" containerName="payload-pre-processing" Sep 10 13:48:36.373486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373423 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3865aa15-1863-4d2a-955e-031a73369bbc" containerName="payload-pre-processing" Sep 10 13:48:36.373610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373488 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3865aa15-1863-4d2a-955e-031a73369bbc" containerName="payload-pre-processing" Sep 10 13:48:36.373610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373501 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.373610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.373506 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a47e330-0507-4521-b593-9c7c69da6341" containerName="istio-proxy" Sep 10 13:48:36.378060 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.378039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:36.382446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.382426 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:48:36.402581 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.402557 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7"] Sep 10 13:48:36.423989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.423958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-21316764-r59r7\" (UID: \"adf64c94-6edd-4108-8dd8-847f913c2245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:36.524581 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.524555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-21316764-r59r7\" (UID: \"adf64c94-6edd-4108-8dd8-847f913c2245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:36.525191 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.525173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-21316764-r59r7\" (UID: \"adf64c94-6edd-4108-8dd8-847f913c2245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:36.688927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.688865 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:36.961776 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:36.961748 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7"] Sep 10 13:48:36.962878 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:48:36.962844 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf64c94_6edd_4108_8dd8_847f913c2245.slice/crio-51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab WatchSource:0}: Error finding container 51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab: Status 404 returned error can't find the container with id 51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab Sep 10 13:48:37.340097 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:37.340065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:37.423714 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:37.423676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" event={"ID":"adf64c94-6edd-4108-8dd8-847f913c2245","Type":"ContainerStarted","Data":"fba984d2d962f579026fb75242ab92b562d9b230748c7dd936a9fdd25069f88d"} Sep 10 13:48:37.424085 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:37.423733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" event={"ID":"adf64c94-6edd-4108-8dd8-847f913c2245","Type":"ContainerStarted","Data":"51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab"} Sep 10 13:48:37.447527 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:37.447482 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" podStartSLOduration=1.447468707 podStartE2EDuration="1.447468707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:37.444393516 +0000 UTC m=+1093.054178431" watchObservedRunningTime="2026-09-10 13:48:37.447468707 +0000 UTC m=+1093.057253614" Sep 10 13:48:38.207342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:38.207301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:38.431940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:38.431903 2559 generic.go:356] "Generic (PLEG): container finished" podID="adf64c94-6edd-4108-8dd8-847f913c2245" containerID="fba984d2d962f579026fb75242ab92b562d9b230748c7dd936a9fdd25069f88d" exitCode=0 Sep 10 13:48:38.432301 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:38.431995 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" event={"ID":"adf64c94-6edd-4108-8dd8-847f913c2245","Type":"ContainerDied","Data":"fba984d2d962f579026fb75242ab92b562d9b230748c7dd936a9fdd25069f88d"} Sep 10 13:48:39.175189 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:39.175148 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:39.584799 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:39.584768 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:39.661470 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:39.661438 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca\") pod \"adf64c94-6edd-4108-8dd8-847f913c2245\" (UID: \"adf64c94-6edd-4108-8dd8-847f913c2245\") " Sep 10 13:48:39.661814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:39.661794 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adf64c94-6edd-4108-8dd8-847f913c2245" (UID: "adf64c94-6edd-4108-8dd8-847f913c2245"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:39.762876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:39.762853 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf64c94-6edd-4108-8dd8-847f913c2245-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:40.173148 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:40.173063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:40.446138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:40.446059 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" Sep 10 13:48:40.446138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:40.446083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7" event={"ID":"adf64c94-6edd-4108-8dd8-847f913c2245","Type":"ContainerDied","Data":"51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab"} Sep 10 13:48:40.446138 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:40.446132 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c25729e866bf145f192992a85b752210db7c4e62768304eeec7958cf77ecab" Sep 10 13:48:42.231775 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:42.231739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:46.985779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:46.985730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:47.075331 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.075301 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:47.075572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.075544 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" podUID="231c056c-3ca0-4b84-9750-03b89af3bb4b" containerName="payload-processing" containerID="cri-o://774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75" gracePeriod=30 Sep 10 13:48:47.086809 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.086780 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:47.087041 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.087013 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" podUID="635226e5-7ead-40e5-b628-7f8d90b8b659" containerName="payload-pre-processing" containerID="cri-o://63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6" gracePeriod=30 Sep 10 13:48:47.129208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.129179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:47.326653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.326617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:47.364677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.364656 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:47.383001 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.382982 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:47.426559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.426531 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74trt\" (UniqueName: \"kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt\") pod \"635226e5-7ead-40e5-b628-7f8d90b8b659\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " Sep 10 13:48:47.426690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.426602 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sgqq\" (UniqueName: \"kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq\") pod \"231c056c-3ca0-4b84-9750-03b89af3bb4b\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " Sep 10 13:48:47.426690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.426637 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume\") pod \"635226e5-7ead-40e5-b628-7f8d90b8b659\" (UID: \"635226e5-7ead-40e5-b628-7f8d90b8b659\") " Sep 10 13:48:47.426823 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.426691 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume\") pod \"231c056c-3ca0-4b84-9750-03b89af3bb4b\" (UID: \"231c056c-3ca0-4b84-9750-03b89af3bb4b\") " Sep 10 13:48:47.427505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.427054 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "635226e5-7ead-40e5-b628-7f8d90b8b659" (UID: "635226e5-7ead-40e5-b628-7f8d90b8b659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:47.427505 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.427464 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "231c056c-3ca0-4b84-9750-03b89af3bb4b" (UID: "231c056c-3ca0-4b84-9750-03b89af3bb4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:47.428590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.428336 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635226e5-7ead-40e5-b628-7f8d90b8b659-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:47.428590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.428362 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231c056c-3ca0-4b84-9750-03b89af3bb4b-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:47.429474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.429443 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt" (OuterVolumeSpecName: "kube-api-access-74trt") pod "635226e5-7ead-40e5-b628-7f8d90b8b659" (UID: "635226e5-7ead-40e5-b628-7f8d90b8b659"). InnerVolumeSpecName "kube-api-access-74trt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:47.430424 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.429866 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq" (OuterVolumeSpecName: "kube-api-access-7sgqq") pod "231c056c-3ca0-4b84-9750-03b89af3bb4b" (UID: "231c056c-3ca0-4b84-9750-03b89af3bb4b"). InnerVolumeSpecName "kube-api-access-7sgqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:48:47.510908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.510868 2559 generic.go:356] "Generic (PLEG): container finished" podID="635226e5-7ead-40e5-b628-7f8d90b8b659" containerID="63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6" exitCode=0 Sep 10 13:48:47.510908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.510921 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" Sep 10 13:48:47.511218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.510950 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" event={"ID":"635226e5-7ead-40e5-b628-7f8d90b8b659","Type":"ContainerDied","Data":"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6"} Sep 10 13:48:47.511218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.511051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb" event={"ID":"635226e5-7ead-40e5-b628-7f8d90b8b659","Type":"ContainerDied","Data":"ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea"} Sep 10 13:48:47.511218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.511087 2559 scope.go:117] "RemoveContainer" containerID="63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6" Sep 10 13:48:47.512642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.512620 2559 generic.go:356] "Generic (PLEG): container finished" podID="231c056c-3ca0-4b84-9750-03b89af3bb4b" containerID="774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75" exitCode=0 Sep 10 13:48:47.512773 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.512695 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" Sep 10 13:48:47.512850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.512774 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" event={"ID":"231c056c-3ca0-4b84-9750-03b89af3bb4b","Type":"ContainerDied","Data":"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75"} Sep 10 13:48:47.512850 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.512827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n" event={"ID":"231c056c-3ca0-4b84-9750-03b89af3bb4b","Type":"ContainerDied","Data":"fda7766f71591cdbcadef14555843dda24002b686a34ea412973047803779f8a"} Sep 10 13:48:47.526569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.526551 2559 scope.go:117] "RemoveContainer" containerID="63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6" Sep 10 13:48:47.526887 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:48:47.526861 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6\": container with ID starting with 63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6 not found: ID does not exist" containerID="63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6" Sep 10 13:48:47.526962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.526895 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6"} err="failed to get container status \"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6\": rpc error: code = NotFound desc = could not find container \"63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6\": container with ID starting with 63c0e5c07e91686ea95ae71136b9c986133f7fa1c8137bde54a4e0314a4d0bb6 not found: ID does not exist" Sep 10 13:48:47.526962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.526912 2559 scope.go:117] "RemoveContainer" containerID="774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75" Sep 10 13:48:47.528979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.528958 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sgqq\" (UniqueName: \"kubernetes.io/projected/231c056c-3ca0-4b84-9750-03b89af3bb4b-kube-api-access-7sgqq\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:47.529065 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.528980 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74trt\" (UniqueName: \"kubernetes.io/projected/635226e5-7ead-40e5-b628-7f8d90b8b659-kube-api-access-74trt\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:47.536051 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:48:47.536030 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635226e5_7ead_40e5_b628_7f8d90b8b659.slice/crio-ef584c908e9fde24ff142e4d1ebc918ed62ce6f5e98a402e8c5136649824bbea\": RecentStats: unable to find data in memory cache]" Sep 10 13:48:47.536264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.536248 2559 scope.go:117] "RemoveContainer" containerID="774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75" Sep 10 13:48:47.536543 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:48:47.536525 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75\": container with ID starting with 774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75 not found: ID does not exist" containerID="774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75" Sep 10 13:48:47.536629 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.536549 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75"} err="failed to get container status \"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75\": rpc error: code = NotFound desc = could not find container \"774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75\": container with ID starting with 774420c90c4344c24e4f36fa1d224ab49add6a17062e91dc2e0c3bbaa598cb75 not found: ID does not exist" Sep 10 13:48:47.555926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.555906 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:47.558342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.558321 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21316764-b89df7f66-p76nb"] Sep 10 13:48:47.572873 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.572850 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:47.579253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.579232 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21316764-6c95fdc4fc-6sj2n"] Sep 10 13:48:47.587523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.587505 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7"] Sep 10 13:48:47.591009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.590992 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-21316764-r59r7"] Sep 10 13:48:47.657587 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.657564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:47.759118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:47.759094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:48.046809 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.046777 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:48.145589 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.145555 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:48.918664 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.918633 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="231c056c-3ca0-4b84-9750-03b89af3bb4b" path="/var/lib/kubelet/pods/231c056c-3ca0-4b84-9750-03b89af3bb4b/volumes" Sep 10 13:48:48.918996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.918983 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635226e5-7ead-40e5-b628-7f8d90b8b659" path="/var/lib/kubelet/pods/635226e5-7ead-40e5-b628-7f8d90b8b659/volumes" Sep 10 13:48:48.919274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.919262 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf64c94-6edd-4108-8dd8-847f913c2245" path="/var/lib/kubelet/pods/adf64c94-6edd-4108-8dd8-847f913c2245/volumes" Sep 10 13:48:48.925991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:48.925968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:49.763027 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:49.762988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:50.484061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:50.484028 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:48:50.878923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:50.878881 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:51.938308 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:51.938271 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:52.057980 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:52.057947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:52.292481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:52.292444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:52.436622 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:52.436592 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:53.251084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251049 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2"] Sep 10 13:48:53.251494 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251480 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635226e5-7ead-40e5-b628-7f8d90b8b659" containerName="payload-pre-processing" Sep 10 13:48:53.251494 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251495 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="635226e5-7ead-40e5-b628-7f8d90b8b659" containerName="payload-pre-processing" Sep 10 13:48:53.251576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251508 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="231c056c-3ca0-4b84-9750-03b89af3bb4b" containerName="payload-processing" Sep 10 13:48:53.251576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251514 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="231c056c-3ca0-4b84-9750-03b89af3bb4b" containerName="payload-processing" Sep 10 13:48:53.251576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251523 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf64c94-6edd-4108-8dd8-847f913c2245" containerName="revoke-keys" Sep 10 13:48:53.251576 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251530 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf64c94-6edd-4108-8dd8-847f913c2245" containerName="revoke-keys" Sep 10 13:48:53.251701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251592 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="231c056c-3ca0-4b84-9750-03b89af3bb4b" containerName="payload-processing" Sep 10 13:48:53.251701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251601 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="635226e5-7ead-40e5-b628-7f8d90b8b659" containerName="payload-pre-processing" Sep 10 13:48:53.251701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.251610 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf64c94-6edd-4108-8dd8-847f913c2245" containerName="revoke-keys" Sep 10 13:48:53.254795 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.254777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:53.257770 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.257746 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h95p6\"" Sep 10 13:48:53.265021 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.264994 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2"] Sep 10 13:48:53.340267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.340235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:53.379488 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.379463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2\" (UID: \"5aaacb07-2d4f-45b0-b34d-30336de57620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:53.479852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.479829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2\" (UID: \"5aaacb07-2d4f-45b0-b34d-30336de57620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:53.480422 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.480405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2\" (UID: \"5aaacb07-2d4f-45b0-b34d-30336de57620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:53.566579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.566544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:53.594754 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.594723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:53.846944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:53.846913 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2"] Sep 10 13:48:53.848216 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:48:53.848195 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaacb07_2d4f_45b0_b34d_30336de57620.slice/crio-32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd WatchSource:0}: Error finding container 32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd: Status 404 returned error can't find the container with id 32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd Sep 10 13:48:54.561212 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:54.561181 2559 generic.go:356] "Generic (PLEG): container finished" podID="5aaacb07-2d4f-45b0-b34d-30336de57620" containerID="4052b7e257abc8d8b0ac43bec3fd5e1ef429fcd2234f808958ab84a1e1452d21" exitCode=0 Sep 10 13:48:54.561529 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:54.561266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" event={"ID":"5aaacb07-2d4f-45b0-b34d-30336de57620","Type":"ContainerDied","Data":"4052b7e257abc8d8b0ac43bec3fd5e1ef429fcd2234f808958ab84a1e1452d21"} Sep 10 13:48:54.561529 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:54.561318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" event={"ID":"5aaacb07-2d4f-45b0-b34d-30336de57620","Type":"ContainerStarted","Data":"32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd"} Sep 10 13:48:54.858203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:54.858177 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:55.701006 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:55.700983 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:55.798579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:55.798551 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca\") pod \"5aaacb07-2d4f-45b0-b34d-30336de57620\" (UID: \"5aaacb07-2d4f-45b0-b34d-30336de57620\") " Sep 10 13:48:55.798892 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:55.798869 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5aaacb07-2d4f-45b0-b34d-30336de57620" (UID: "5aaacb07-2d4f-45b0-b34d-30336de57620"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:48:55.899980 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:55.899924 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaacb07-2d4f-45b0-b34d-30336de57620-maas-api-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:48:56.086004 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:56.085953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:56.574704 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:56.574672 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" Sep 10 13:48:56.574846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:56.574712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2" event={"ID":"5aaacb07-2d4f-45b0-b34d-30336de57620","Type":"ContainerDied","Data":"32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd"} Sep 10 13:48:56.574846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:56.574756 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a1d3d074f716a93611684fe4ff7ac56189469bb48870b4729edcb86b9180cd" Sep 10 13:48:57.204425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.204394 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:57.313256 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.313213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:57.406153 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.406108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:57.546217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.546184 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:48:57.546451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.546425 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" containerID="cri-o://34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de" gracePeriod=30 Sep 10 13:48:57.550261 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.550232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:57.656884 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:57.656849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:58.041905 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:58.041866 2559 patch_prober.go:28] interesting pod/e2e-mt-21316764-openshift-default-fb968c688-gfrtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:48:58.042079 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:58.041926 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:48:58.098427 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:58.098359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:48:58.204094 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:48:58.204058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:02.001513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.001478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:49:02.805459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.805436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:49:02.862885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862816 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.862885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862849 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.862885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862867 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862893 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862934 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.862962 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863023 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82p5q\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863099 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token\") pod \"35bcfb10-5224-4481-a870-2828ae52aff2\" (UID: \"35bcfb10-5224-4481-a870-2828ae52aff2\") " Sep 10 13:49:02.863128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863105 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:02.863533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863150 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:02.863533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:02.863533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863458 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.863533 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863477 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.863752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863559 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:02.863752 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.863582 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data" (OuterVolumeSpecName: "istio-data") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:02.865713 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.865665 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:02.865713 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.865669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q" (OuterVolumeSpecName: "kube-api-access-82p5q") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "kube-api-access-82p5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:02.865894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.865772 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:49:02.865894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.865779 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token" (OuterVolumeSpecName: "istio-token") pod "35bcfb10-5224-4481-a870-2828ae52aff2" (UID: "35bcfb10-5224-4481-a870-2828ae52aff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:02.964235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964207 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcfb10-5224-4481-a870-2828ae52aff2-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964231 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82p5q\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-kube-api-access-82p5q\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964235 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964241 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcfb10-5224-4481-a870-2828ae52aff2-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964249 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964258 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964265 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcfb10-5224-4481-a870-2828ae52aff2-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:02.964467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:02.964273 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcfb10-5224-4481-a870-2828ae52aff2-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:03.624756 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.624725 2559 generic.go:356] "Generic (PLEG): container finished" podID="35bcfb10-5224-4481-a870-2828ae52aff2" containerID="34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de" exitCode=0 Sep 10 13:49:03.625163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.624773 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" Sep 10 13:49:03.625163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.624814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerDied","Data":"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de"} Sep 10 13:49:03.625163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.624871 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn" event={"ID":"35bcfb10-5224-4481-a870-2828ae52aff2","Type":"ContainerDied","Data":"4626dba282e12cd1871ef331ac5063599d96d86e2c238862aae19aca1a565b82"} Sep 10 13:49:03.625163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.624893 2559 scope.go:117] "RemoveContainer" containerID="34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de" Sep 10 13:49:03.636501 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.636482 2559 scope.go:117] "RemoveContainer" containerID="96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae" Sep 10 13:49:03.644931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.644912 2559 scope.go:117] "RemoveContainer" containerID="34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de" Sep 10 13:49:03.645214 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:03.645196 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de\": container with ID starting with 34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de not found: ID does not exist" containerID="34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de" Sep 10 13:49:03.645266 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.645223 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de"} err="failed to get container status \"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de\": rpc error: code = NotFound desc = could not find container \"34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de\": container with ID starting with 34ef59d0ce690fde98f94d07ed65f77e78eb36ab3c45ca1923b2918713eb23de not found: ID does not exist" Sep 10 13:49:03.645266 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.645242 2559 scope.go:117] "RemoveContainer" containerID="96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae" Sep 10 13:49:03.645542 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:03.645519 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae\": container with ID starting with 96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae not found: ID does not exist" containerID="96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae" Sep 10 13:49:03.645610 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.645549 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae"} err="failed to get container status \"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae\": rpc error: code = NotFound desc = could not find container \"96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae\": container with ID starting with 96bf51fe975de064aac2eb8e0e503f74be44396c7f876309e4cad506e85e0dae not found: ID does not exist" Sep 10 13:49:03.661958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.661934 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:49:03.675688 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:03.675667 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21316764-openshift-default-fb968c688-gfrtn"] Sep 10 13:49:04.257869 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.257837 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr"] Sep 10 13:49:04.262733 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.262708 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f1d168-29817465-t6npr"] Sep 10 13:49:04.281034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.281004 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:04.394006 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.393973 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:04.515777 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.515696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:04.629205 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.629173 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2"] Sep 10 13:49:04.631566 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.631545 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f1d168-xhcp2"] Sep 10 13:49:04.919786 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.919706 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" path="/var/lib/kubelet/pods/35bcfb10-5224-4481-a870-2828ae52aff2/volumes" Sep 10 13:49:04.920069 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.920054 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aaacb07-2d4f-45b0-b34d-30336de57620" path="/var/lib/kubelet/pods/5aaacb07-2d4f-45b0-b34d-30336de57620/volumes" Sep 10 13:49:04.920311 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:04.920299 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f739f0-9b14-42f7-8ad7-67cd3a5b3668" path="/var/lib/kubelet/pods/93f739f0-9b14-42f7-8ad7-67cd3a5b3668/volumes" Sep 10 13:49:05.331284 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:05.331247 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:05.427062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:05.427033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:06.539469 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:06.539437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:06.667209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:06.667163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:06.798831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:06.798736 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:06.902818 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:06.902790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:07.068195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:07.068108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:11.091089 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:11.091052 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:49:12.170532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:12.170502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:12.251268 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:12.251235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:12.441900 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:12.441822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:49:12.442100 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:12.442073 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" containerID="cri-o://ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289" gracePeriod=30 Sep 10 13:49:12.480474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:12.480438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:13.762161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:13.762125 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:14.123030 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:14.122914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:14.220092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:14.220053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:14.781057 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:14.781013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:15.141539 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:15.141445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:15.584554 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:15.584517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:16.044440 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:16.044402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:17.696729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.696699 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:49:17.718852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.718776 2559 generic.go:356] "Generic (PLEG): container finished" podID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerID="ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289" exitCode=0 Sep 10 13:49:17.718852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.718818 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" Sep 10 13:49:17.719012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.718851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerDied","Data":"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289"} Sep 10 13:49:17.719012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.718901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2" event={"ID":"7254af13-d1f5-4022-bfb2-71dd0086a4ae","Type":"ContainerDied","Data":"6587e5ba6c7dc6925b2c63a97409508371c604001bd42597a55bae00053fcc61"} Sep 10 13:49:17.719012 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.718924 2559 scope.go:117] "RemoveContainer" containerID="ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289" Sep 10 13:49:17.729855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.729828 2559 scope.go:117] "RemoveContainer" containerID="b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e" Sep 10 13:49:17.740969 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.740946 2559 scope.go:117] "RemoveContainer" containerID="ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289" Sep 10 13:49:17.741361 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:17.741337 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289\": container with ID starting with ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289 not found: ID does not exist" containerID="ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289" Sep 10 13:49:17.741502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.741395 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289"} err="failed to get container status \"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289\": rpc error: code = NotFound desc = could not find container \"ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289\": container with ID starting with ef1a7d83b547c4801e357e717351a601b086f7938264c335027987e901b11289 not found: ID does not exist" Sep 10 13:49:17.741502 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.741422 2559 scope.go:117] "RemoveContainer" containerID="b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e" Sep 10 13:49:17.741744 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:17.741719 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e\": container with ID starting with b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e not found: ID does not exist" containerID="b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e" Sep 10 13:49:17.741813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.741752 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e"} err="failed to get container status \"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e\": rpc error: code = NotFound desc = could not find container \"b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e\": container with ID starting with b01dca2ad6b04690491f470d7b6ba01b9e53448bc76d6f36480f5934f10ea61e not found: ID does not exist" Sep 10 13:49:17.806020 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.805982 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806053 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806096 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806123 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806184 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806246 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806272 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806240 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:17.806607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5vr2\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2\") pod \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\" (UID: \"7254af13-d1f5-4022-bfb2-71dd0086a4ae\") " Sep 10 13:49:17.806607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:17.806697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:17.806697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806640 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:17.806814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806723 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.806814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806743 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.806814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806757 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.806968 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.806813 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:17.808107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.808086 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:49:17.808462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.808435 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2" (OuterVolumeSpecName: "kube-api-access-r5vr2") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "kube-api-access-r5vr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:17.808642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.808623 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:17.808720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.808693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7254af13-d1f5-4022-bfb2-71dd0086a4ae" (UID: "7254af13-d1f5-4022-bfb2-71dd0086a4ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:17.907915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907886 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.907915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907911 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.907915 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907920 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7254af13-d1f5-4022-bfb2-71dd0086a4ae-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.908084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907928 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.908084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907936 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:17.908084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:17.907945 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vr2\" (UniqueName: \"kubernetes.io/projected/7254af13-d1f5-4022-bfb2-71dd0086a4ae-kube-api-access-r5vr2\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:18.045526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:18.045498 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:49:18.050536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:18.050511 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f1d168-openshift-default-5d876c6db-x4xv2"] Sep 10 13:49:18.405640 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:18.405533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:18.918777 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:18.918731 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" path="/var/lib/kubelet/pods/7254af13-d1f5-4022-bfb2-71dd0086a4ae/volumes" Sep 10 13:49:19.079122 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:19.079084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:19.176406 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:19.176301 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:19.340266 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:19.340232 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:20.544078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:20.544027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:20.644346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:20.644308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:20.752562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:20.752525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:21.117395 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.117349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:21.212743 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.212706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:21.301992 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.301959 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:49:21.302393 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302379 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:21.302443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302395 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:21.302443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302402 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aaacb07-2d4f-45b0-b34d-30336de57620" containerName="revoke-keys" Sep 10 13:49:21.302443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302408 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aaacb07-2d4f-45b0-b34d-30336de57620" containerName="revoke-keys" Sep 10 13:49:21.302443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302425 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:21.302443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302431 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:21.302604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302492 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aaacb07-2d4f-45b0-b34d-30336de57620" containerName="revoke-keys" Sep 10 13:49:21.302604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302503 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:21.302604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302509 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:21.302604 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.302516 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:21.307218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.307197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.309984 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.309964 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-3d0caa-dockercfg-xghjr\"" Sep 10 13:49:21.310061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.310003 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-3d0caa\"" Sep 10 13:49:21.316572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.316549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:49:21.339068 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.339035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.339894 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.339361 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbpc\" (UniqueName: \"kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.440194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.440112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.440194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.440189 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbpc\" (UniqueName: \"kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.440916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.440895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.450976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.450952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbpc\" (UniqueName: \"kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc\") pod \"payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.618467 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.618434 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:21.908807 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:21.908782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:49:21.910668 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:49:21.910638 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff571aa_bbf8_4010_8bd0_e0001e9a34fd.slice/crio-a8b61efd7d9975e5042e7f1ed3a8328667a5b11bb045f670fd4d84f187e50d3e WatchSource:0}: Error finding container a8b61efd7d9975e5042e7f1ed3a8328667a5b11bb045f670fd4d84f187e50d3e: Status 404 returned error can't find the container with id a8b61efd7d9975e5042e7f1ed3a8328667a5b11bb045f670fd4d84f187e50d3e Sep 10 13:49:22.754792 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:22.754758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" event={"ID":"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd","Type":"ContainerStarted","Data":"e485ccffc3ed3ed6eccac3afbd5c8f8f630b7d52c683375a0d2b40e218d28fc2"} Sep 10 13:49:22.755177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:22.754819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" event={"ID":"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd","Type":"ContainerStarted","Data":"a8b61efd7d9975e5042e7f1ed3a8328667a5b11bb045f670fd4d84f187e50d3e"} Sep 10 13:49:22.755177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:22.754842 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:22.778067 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:22.778022 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" podStartSLOduration=1.7779991609999999 podStartE2EDuration="1.777999161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:22.774036983 +0000 UTC m=+1138.383821896" watchObservedRunningTime="2026-09-10 13:49:22.777999161 +0000 UTC m=+1138.387784129" Sep 10 13:49:23.418386 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:23.418335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:23.518252 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:23.518214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:23.617329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:23.617287 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:23.705095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:23.705013 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:23.775912 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:23.775884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:24.067010 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:24.066965 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:24.211835 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:24.211806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:24.500215 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:24.500135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:49:25.670760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:25.670731 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:31.217226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:31.217189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:31.315757 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:31.315716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:31.511753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:31.511724 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:49:31.511969 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:31.511948 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" containerID="cri-o://3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17" gracePeriod=30 Sep 10 13:49:31.553137 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:31.553105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:32.102099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.102060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:32.399976 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.399903 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 13:49:32.400507 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.400490 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:32.400584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.400509 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:32.400584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.400543 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:32.400584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.400552 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7254af13-d1f5-4022-bfb2-71dd0086a4ae" containerName="istio-proxy" Sep 10 13:49:32.400677 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.400649 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bcfb10-5224-4481-a870-2828ae52aff2" containerName="istio-proxy" Sep 10 13:49:32.405484 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.405459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.412039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.411808 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-579ae1-openshift-default-dockercfg-8qg8v\"" Sep 10 13:49:32.420714 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.420692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 13:49:32.422061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.422034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:32.433177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gj5\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.433678 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.433578 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.534940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.534905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gj5\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535102 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.534945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535102 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535102 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.535853 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.535832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.537518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.537495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.537706 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.537687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.550815 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.550794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gj5\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.551739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.551718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token\") pod \"e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:32.554694 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.554652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:32.715439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:32.715359 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:33.010695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.010667 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 13:49:33.012408 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:49:33.012363 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de37bf8_5152_418d_abf8_64516c1bc578.slice/crio-44b89059f8a96e06811084f7761de3bae887129af2550cdbca9fb3a1d3522e14 WatchSource:0}: Error finding container 44b89059f8a96e06811084f7761de3bae887129af2550cdbca9fb3a1d3522e14: Status 404 returned error can't find the container with id 44b89059f8a96e06811084f7761de3bae887129af2550cdbca9fb3a1d3522e14 Sep 10 13:49:33.014454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.014424 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:33.014523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.014486 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:33.014523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.014514 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:33.763735 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.763703 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:49:33.832279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.832249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerStarted","Data":"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80"} Sep 10 13:49:33.832279 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.832293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerStarted","Data":"44b89059f8a96e06811084f7761de3bae887129af2550cdbca9fb3a1d3522e14"} Sep 10 13:49:33.859953 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:33.859911 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podStartSLOduration=1.859895353 podStartE2EDuration="1.859895353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:33.857335923 +0000 UTC m=+1149.467120836" watchObservedRunningTime="2026-09-10 13:49:33.859895353 +0000 UTC m=+1149.469680265" Sep 10 13:49:34.058613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.058526 2559 patch_prober.go:28] interesting pod/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 13:49:34.058613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.058576 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 13:49:34.716181 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.716148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:34.721086 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.721062 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:34.839346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.839318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:34.840061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:34.840040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:35.522227 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:35.522187 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:35.905907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:35.905821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:35.999295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:35.999264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:36.760444 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.760420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:49:36.852810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.852737 2559 generic.go:356] "Generic (PLEG): container finished" podID="b327f38b-8381-4416-b35a-fe179cdaa842" containerID="3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17" exitCode=0 Sep 10 13:49:36.852810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.852798 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" Sep 10 13:49:36.853017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.852861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerDied","Data":"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17"} Sep 10 13:49:36.853017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.852952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p" event={"ID":"b327f38b-8381-4416-b35a-fe179cdaa842","Type":"ContainerDied","Data":"15440cef2230f41fe56a2c4683dbbdc5709cdf6a0f3f0dc96044c8ceb56cd900"} Sep 10 13:49:36.853017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.852986 2559 scope.go:117] "RemoveContainer" containerID="3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17" Sep 10 13:49:36.857193 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.857165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:36.862568 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.862551 2559 scope.go:117] "RemoveContainer" containerID="2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365" Sep 10 13:49:36.870214 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870194 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870256 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk24g\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870306 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870283 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870324 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870380 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870414 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870434 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870469 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy\") pod \"b327f38b-8381-4416-b35a-fe179cdaa842\" (UID: \"b327f38b-8381-4416-b35a-fe179cdaa842\") " Sep 10 13:49:36.870747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870652 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:36.870747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870698 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:36.870916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870692 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data" (OuterVolumeSpecName: "istio-data") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:36.870916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:49:36.870916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870866 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.870916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870888 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.870916 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.870902 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.871176 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.871085 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:36.871478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.871456 2559 scope.go:117] "RemoveContainer" containerID="3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17" Sep 10 13:49:36.871810 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:36.871784 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17\": container with ID starting with 3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17 not found: ID does not exist" containerID="3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17" Sep 10 13:49:36.871953 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.871818 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17"} err="failed to get container status \"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17\": rpc error: code = NotFound desc = could not find container \"3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17\": container with ID starting with 3ad15cb421032eb5ddd8eae8031efddd31081999990bb708f9946fcaf81f9c17 not found: ID does not exist" Sep 10 13:49:36.871953 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.871843 2559 scope.go:117] "RemoveContainer" containerID="2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365" Sep 10 13:49:36.872256 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:36.872226 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365\": container with ID starting with 2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365 not found: ID does not exist" containerID="2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365" Sep 10 13:49:36.872347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.872262 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365"} err="failed to get container status \"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365\": rpc error: code = NotFound desc = could not find container \"2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365\": container with ID starting with 2cc14d7432de2dc59ec69b55be629e6999b1c222b319eb602231284ad5798365 not found: ID does not exist" Sep 10 13:49:36.872917 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.872895 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:49:36.873007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.872926 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token" (OuterVolumeSpecName: "istio-token") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:36.873007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.872947 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g" (OuterVolumeSpecName: "kube-api-access-tk24g") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "kube-api-access-tk24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:49:36.873007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.872967 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b327f38b-8381-4416-b35a-fe179cdaa842" (UID: "b327f38b-8381-4416-b35a-fe179cdaa842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:49:36.971814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971789 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.971814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971813 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.972208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971822 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327f38b-8381-4416-b35a-fe179cdaa842-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.972208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971831 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327f38b-8381-4416-b35a-fe179cdaa842-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.972208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971844 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk24g\" (UniqueName: \"kubernetes.io/projected/b327f38b-8381-4416-b35a-fe179cdaa842-kube-api-access-tk24g\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:36.972208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:36.971856 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327f38b-8381-4416-b35a-fe179cdaa842-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:49:37.185282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:37.185222 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:49:37.190177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:37.190147 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e6d037-openshift-default-7fc5568d7d-46l5p"] Sep 10 13:49:37.845572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:37.845541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:38.184399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184304 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 13:49:38.184761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184717 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.184761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184730 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.184761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184747 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.184761 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184753 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.184908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184825 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.184908 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.184835 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" containerName="istio-proxy" Sep 10 13:49:38.188404 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.188385 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.191587 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.191565 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-97x99\"" Sep 10 13:49:38.191718 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.191681 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-579ae1-serving-cert\"" Sep 10 13:49:38.192558 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.192539 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 13:49:38.204653 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.204630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 13:49:38.230575 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.230547 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:38.282118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.282093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.282233 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.282131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmv9\" (UniqueName: \"kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.282233 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.282218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.282305 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.282280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.335855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.335829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:38.383051 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.383026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.383162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.383077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.383162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.383093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmv9\" (UniqueName: \"kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.383162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.383127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.383292 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:38.383179 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-serving-cert: secret "maas-api-e2e-shared-a-w3-579ae1-serving-cert" not found Sep 10 13:49:38.383292 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:49:38.383261 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls podName:e0855d13-a1c0-4e38-9c2b-2134f534796b nodeName:}" failed. No retries permitted until 2026-09-10 13:49:38.883239206 +0000 UTC m=+1154.493024097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" (UID: "e0855d13-a1c0-4e38-9c2b-2134f534796b") : secret "maas-api-e2e-shared-a-w3-579ae1-serving-cert" not found Sep 10 13:49:38.383759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.383736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.385607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.385587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.396267 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.396245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmv9\" (UniqueName: \"kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.745756 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.745729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:38.888179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.888150 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.890495 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.890475 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:38.918553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.918533 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b327f38b-8381-4416-b35a-fe179cdaa842" path="/var/lib/kubelet/pods/b327f38b-8381-4416-b35a-fe179cdaa842/volumes" Sep 10 13:49:38.951688 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:38.951658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:39.102547 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.102521 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:39.201784 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.201752 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:39.312644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.312609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:39.411514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.411490 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 13:49:39.412787 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:49:39.412759 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0855d13_a1c0_4e38_9c2b_2134f534796b.slice/crio-68220f327bc1b5149290cd20b751951fec2407326097e17f132f0e384ed08188 WatchSource:0}: Error finding container 68220f327bc1b5149290cd20b751951fec2407326097e17f132f0e384ed08188: Status 404 returned error can't find the container with id 68220f327bc1b5149290cd20b751951fec2407326097e17f132f0e384ed08188 Sep 10 13:49:39.414198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.414180 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:49:39.672574 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.672493 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:49:39.677296 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.677272 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.682230 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.682210 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b3723e-openshift-default-dockercfg-2czxw\"" Sep 10 13:49:39.697797 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.697777 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:49:39.796898 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.796857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797070 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.796913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797070 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.796970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797070 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9zr\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.797331 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.797219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.875324 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.875296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" event={"ID":"e0855d13-a1c0-4e38-9c2b-2134f534796b","Type":"ContainerStarted","Data":"68220f327bc1b5149290cd20b751951fec2407326097e17f132f0e384ed08188"} Sep 10 13:49:39.898188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9zr\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898525 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898646 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898690 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898774 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898774 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.898909 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.898890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.899079 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.899046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.899171 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.899150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.900808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.900780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.900808 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.900804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.925466 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.925359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.925886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.925863 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9zr\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr\") pod \"e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:39.989112 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:39.988961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:40.324814 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:49:40.324781 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf587d2_c502_480c_b40f_7bc25c60824e.slice/crio-0d799a8c07e38da23420af55b1a6bed3fc4378d546a35364b1f5f1ec4c157899 WatchSource:0}: Error finding container 0d799a8c07e38da23420af55b1a6bed3fc4378d546a35364b1f5f1ec4c157899: Status 404 returned error can't find the container with id 0d799a8c07e38da23420af55b1a6bed3fc4378d546a35364b1f5f1ec4c157899 Sep 10 13:49:40.324814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.324801 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:49:40.327388 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.327341 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:40.327490 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.327422 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:40.327490 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.327462 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:40.355624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.355597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:40.478660 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.478605 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:40.886682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.886644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerStarted","Data":"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b"} Sep 10 13:49:40.886871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.886707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerStarted","Data":"0d799a8c07e38da23420af55b1a6bed3fc4378d546a35364b1f5f1ec4c157899"} Sep 10 13:49:40.924628 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.924579 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podStartSLOduration=1.924563847 podStartE2EDuration="1.924563847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:40.92180146 +0000 UTC m=+1156.531586374" watchObservedRunningTime="2026-09-10 13:49:40.924563847 +0000 UTC m=+1156.534348759" Sep 10 13:49:40.989523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:40.989491 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:41.813299 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:41.813261 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:41.994147 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:41.994119 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:42.904657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:42.904619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" event={"ID":"e0855d13-a1c0-4e38-9c2b-2134f534796b","Type":"ContainerStarted","Data":"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984"} Sep 10 13:49:42.905090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:42.904717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:42.905090 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:42.904773 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:42.905759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:42.905739 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:49:42.933979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:42.933935 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" podStartSLOduration=2.505650105 podStartE2EDuration="4.933921536s" totalImagesPullingTime="2.428271431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:42.930507384 +0000 UTC m=+1158.540292298" watchObservedRunningTime="2026-09-10 13:49:42.933921536 +0000 UTC m=+1158.543706510" Sep 10 13:49:43.120351 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:43.120312 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:43.269886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:43.269848 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:43.617092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:43.617002 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:43.899360 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:43.899282 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:49:45.160588 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:45.160549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:45.260446 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:45.260408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:45.491673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:45.491566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:45.908547 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:45.908489 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:46.008734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:46.008681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:46.695729 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:46.695692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:48.536242 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:48.536161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:48.919522 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:48.919447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 13:49:50.025229 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:50.025192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:49:53.198583 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.198550 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:49:53.987734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.987704 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de37bf8-5152-418d-abf8-64516c1bc578" containerID="2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80" exitCode=0 Sep 10 13:49:53.987912 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.987763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerDied","Data":"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80"} Sep 10 13:49:53.988115 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.988102 2559 scope.go:117] "RemoveContainer" containerID="2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80" Sep 10 13:49:53.989099 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.989072 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:53.989164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.989136 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:53.989210 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:53.989171 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:49:54.995307 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:54.995266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerStarted","Data":"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58"} Sep 10 13:49:55.715879 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:55.715850 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:49:55.716159 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:55.716133 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 13:49:55.716274 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:55.716184 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 13:49:56.715836 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:56.715795 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 13:49:56.716328 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:56.715878 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 13:49:57.716061 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:57.716016 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 13:49:57.716481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:57.716103 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 13:49:58.715745 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:58.715709 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 13:49:58.715929 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:58.715767 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 13:49:58.946584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:58.946514 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 13:49:58.947060 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:58.946609 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 13:49:59.716295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:59.716223 2559 patch_prober.go:28] interesting pod/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Sep 10 13:49:59.716295 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:59.716280 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Sep 10 13:49:59.846219 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:49:59.846006 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:00.032996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.032954 2559 generic.go:356] "Generic (PLEG): container finished" podID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerID="17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b" exitCode=0 Sep 10 13:50:00.033509 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.033021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerDied","Data":"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b"} Sep 10 13:50:00.033509 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.033414 2559 scope.go:117] "RemoveContainer" containerID="17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b" Sep 10 13:50:00.034428 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.034403 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:50:00.034526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.034466 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:50:00.034526 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.034495 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:50:00.784833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.784794 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:50:00.785236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.785217 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:50:00.786161 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:00.786137 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 13:50:01.055830 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:01.055716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerStarted","Data":"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e"} Sep 10 13:50:01.670933 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:01.670900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:01.989478 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:01.989397 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:50:01.989624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:01.989562 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 10 13:50:01.989624 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:01.989605 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 10 13:50:02.727217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:02.727182 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:02.989724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:02.989642 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 10 13:50:02.989724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:02.989696 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 10 13:50:03.989912 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:03.989879 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 10 13:50:03.990437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:03.989973 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 10 13:50:04.377125 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:04.377094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:04.921395 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:04.921345 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:04.989550 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:04.989510 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 10 13:50:04.989737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:04.989571 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 10 13:50:05.989723 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:05.989686 2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 10 13:50:05.990117 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:05.989757 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 10 13:50:06.248148 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.248061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:06.346071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.346041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:06.466875 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.466848 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:06.859579 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.859543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:06.993162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.993134 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:50:06.993654 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.993548 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:50:06.994198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:06.994177 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:50:07.064562 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:07.064536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:07.247698 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:07.247616 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:08.139214 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:08.139179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:08.244404 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:08.244355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:09.229131 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:09.229080 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.367760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:10.367726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.530843 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:10.530796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:10.701846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:10.701772 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:12.183107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.183077 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 13:50:12.188801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.188779 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.210890 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.210868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.211026 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.210906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.211026 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.210924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.211145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.211096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6xkf\" (UniqueName: \"kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.214999 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.214978 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d5c012-serving-cert\"" Sep 10 13:50:12.293883 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.293509 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:12.312207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.312173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6xkf\" (UniqueName: \"kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.312358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.312230 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.312358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.312278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.312358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.312304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.312559 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:50:12.312438 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-serving-cert: secret "maas-api-e2e-shared-b-w3-d5c012-serving-cert" not found Sep 10 13:50:12.312559 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:50:12.312507 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls podName:a4cdaa2d-9abb-4bd5-a649-b6f709680436 nodeName:}" failed. No retries permitted until 2026-09-10 13:50:12.81248789 +0000 UTC m=+1188.422272790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" (UID: "a4cdaa2d-9abb-4bd5-a649-b6f709680436") : secret "maas-api-e2e-shared-b-w3-d5c012-serving-cert" not found Sep 10 13:50:12.313052 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.313027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.314900 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.314880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.324418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.324396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 13:50:12.327413 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.327391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6xkf\" (UniqueName: \"kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.407217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.407193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:12.615472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.615436 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:12.816142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.816115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:12.818277 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:12.818251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:13.102049 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:13.102018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:13.215219 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:13.215180 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:13.370494 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:13.370409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:13.412335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:13.412306 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 13:50:13.417743 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:50:13.417718 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cdaa2d_9abb_4bd5_a649_b6f709680436.slice/crio-190275ae517aded3f00e6c2fa40e3e8155fbaecb07e77b28e65de10d127b7d11 WatchSource:0}: Error finding container 190275ae517aded3f00e6c2fa40e3e8155fbaecb07e77b28e65de10d127b7d11: Status 404 returned error can't find the container with id 190275ae517aded3f00e6c2fa40e3e8155fbaecb07e77b28e65de10d127b7d11 Sep 10 13:50:14.149403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:14.149295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" event={"ID":"a4cdaa2d-9abb-4bd5-a649-b6f709680436","Type":"ContainerStarted","Data":"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443"} Sep 10 13:50:14.149403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:14.149360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" event={"ID":"a4cdaa2d-9abb-4bd5-a649-b6f709680436","Type":"ContainerStarted","Data":"190275ae517aded3f00e6c2fa40e3e8155fbaecb07e77b28e65de10d127b7d11"} Sep 10 13:50:14.149642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:14.149446 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:14.184020 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:14.183973 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" podStartSLOduration=1.8007909130000002 podStartE2EDuration="2.183957918s" totalImagesPullingTime="383.167005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:14.181082959 +0000 UTC m=+1189.790867872" watchObservedRunningTime="2026-09-10 13:50:14.183957918 +0000 UTC m=+1189.793742832" Sep 10 13:50:14.243906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:14.243877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:15.574204 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:15.574161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:16.914239 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:16.914175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:17.605914 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:17.605883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:50:18.157425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:18.157387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:18.434459 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:18.434360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:18.530357 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:18.530319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:19.011701 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:19.011668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:20.161351 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:20.161321 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 13:50:21.039255 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:21.039219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:21.726991 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:21.726958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:21.912571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:21.912538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:22.653110 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:22.653079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:24.253553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:24.253499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:24.981020 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:24.980989 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:50:24.982561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:24.982535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:50:25.049926 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:25.049901 2559 scope.go:117] "RemoveContainer" containerID="b5787cec3989f6f761d520134acc82a8a190eaabebe449b323ee289a85b50712" Sep 10 13:50:25.388956 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:25.388917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:26.778177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:26.778143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:27.809336 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.809304 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:50:27.814408 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.814387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:27.817075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.817053 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a6883ccf-serving-cert\"" Sep 10 13:50:27.830896 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.830875 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:50:27.944410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.944352 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:27.944569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.944440 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:27.944569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.944459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:27.944569 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:27.944548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7x7\" (UniqueName: \"kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.045918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.045886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.046058 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.045926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.046058 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.046019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7x7\" (UniqueName: \"kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.046170 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.046064 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.046230 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:50:28.046199 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-serving-cert: secret "maas-api-e2e-mt-a6883ccf-serving-cert" not found Sep 10 13:50:28.046284 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:50:28.046274 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls podName:b8f826da-514f-4220-9b3c-93d59e376d5e nodeName:}" failed. No retries permitted until 2026-09-10 13:50:28.546242471 +0000 UTC m=+1204.156027368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls") pod "maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" (UID: "b8f826da-514f-4220-9b3c-93d59e376d5e") : secret "maas-api-e2e-mt-a6883ccf-serving-cert" not found Sep 10 13:50:28.046682 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.046662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.048329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.048303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.060685 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.060629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7x7\" (UniqueName: \"kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.225874 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.225849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:28.360703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.360621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:28.550289 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.550260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.552695 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.552668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") pod \"maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.725353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.725264 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:28.726344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.726308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:28.843965 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:28.843905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:29.041280 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:29.041249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:50:29.042109 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:50:29.042073 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f826da_514f_4220_9b3c_93d59e376d5e.slice/crio-30f88f186701305e83097478c12ce28bd839769e03da34d167b62e51d699b541 WatchSource:0}: Error finding container 30f88f186701305e83097478c12ce28bd839769e03da34d167b62e51d699b541: Status 404 returned error can't find the container with id 30f88f186701305e83097478c12ce28bd839769e03da34d167b62e51d699b541 Sep 10 13:50:29.243053 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:29.243020 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" event={"ID":"b8f826da-514f-4220-9b3c-93d59e376d5e","Type":"ContainerStarted","Data":"30f88f186701305e83097478c12ce28bd839769e03da34d167b62e51d699b541"} Sep 10 13:50:29.402063 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:29.401975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.250985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:30.250946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" event={"ID":"b8f826da-514f-4220-9b3c-93d59e376d5e","Type":"ContainerStarted","Data":"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f"} Sep 10 13:50:30.251350 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:30.251058 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:30.284458 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:30.284399 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" podStartSLOduration=2.2721355819999998 podStartE2EDuration="3.284361489s" totalImagesPullingTime="1.012225907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:30.284004736 +0000 UTC m=+1205.893789653" watchObservedRunningTime="2026-09-10 13:50:30.284361489 +0000 UTC m=+1205.894146404" Sep 10 13:50:30.297356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:30.297323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:30.552749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:30.552713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:31.718438 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:31.718395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:32.944040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:32.944011 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:33.076046 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:33.076012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:33.836756 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:33.836717 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:34.746607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:34.746575 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:36.263324 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:36.263296 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:50:36.893418 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:36.893385 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:50:46.140270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.140232 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:50:46.145127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.145106 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.163301 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.163275 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"openshift-service-ca.crt\"" Sep 10 13:50:46.163486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.163336 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"auth-test-model-579ae1-kserve-self-signed-certs\"" Sep 10 13:50:46.164228 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.164205 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"default-dockercfg-rjdk6\"" Sep 10 13:50:46.164358 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.164234 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"kube-root-ca.crt\"" Sep 10 13:50:46.166499 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.166474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:50:46.201607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.201576 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:46.314748 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnhb\" (UniqueName: \"kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.314907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.314907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.314907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.314907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.315071 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.314929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416206 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416206 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnhb\" (UniqueName: \"kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416675 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.416825 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.416715 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.418590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.418559 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.418789 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.418772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.425451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.425426 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnhb\" (UniqueName: \"kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb\") pod \"auth-test-model-579ae1-kserve-cf6794bff-z5h5x\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.455011 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.454991 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:46.731429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:46.731402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:50:46.732916 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:50:46.732884 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode393e8ec_aad8_41d8_9bc2_9533b4cec161.slice/crio-cd55f5e1700452e50f33bcdc9389c63e7670c2a7ce8fdd8748591c699469e67c WatchSource:0}: Error finding container cd55f5e1700452e50f33bcdc9389c63e7670c2a7ce8fdd8748591c699469e67c: Status 404 returned error can't find the container with id cd55f5e1700452e50f33bcdc9389c63e7670c2a7ce8fdd8748591c699469e67c Sep 10 13:50:47.364759 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:47.364717 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" event={"ID":"e393e8ec-aad8-41d8-9bc2-9533b4cec161","Type":"ContainerStarted","Data":"7483c2dc1b865f746e34108a4246e6ba112df557efc922bb4854965e0a0c50e7"} Sep 10 13:50:47.365216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:47.364769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" event={"ID":"e393e8ec-aad8-41d8-9bc2-9533b4cec161","Type":"ContainerStarted","Data":"cd55f5e1700452e50f33bcdc9389c63e7670c2a7ce8fdd8748591c699469e67c"} Sep 10 13:50:47.411768 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:47.411705 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" podStartSLOduration=1.411684827 podStartE2EDuration="1.411684827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:47.406462684 +0000 UTC m=+1223.016247597" watchObservedRunningTime="2026-09-10 13:50:47.411684827 +0000 UTC m=+1223.021469741" Sep 10 13:50:47.448871 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:47.448841 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:49.352959 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:49.352921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:51.726017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:51.725978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:51.831124 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:51.831086 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:53.375098 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:53.374928 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:54.392414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:54.392354 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:50:54.658834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:54.658740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:55.933582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:55.933549 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:56.455766 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:56.455737 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:56.455958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:56.455784 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:56.467905 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:56.467876 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:56.922854 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:56.922823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:50:57.443995 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:57.443964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:50:57.942736 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:50:57.942693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:25.082419 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:25.082333 2559 scope.go:117] "RemoveContainer" containerID="a4c1a13180b601acf4c1dc897a3c45386cbba4824c8b2417de83e5668a7af97a" Sep 10 13:51:25.091443 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:25.091426 2559 scope.go:117] "RemoveContainer" containerID="c0df0fdd4b3dba98cf3803967b6014f01d839a371a66d0897d2c88a33b670565" Sep 10 13:51:25.099849 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:25.099832 2559 scope.go:117] "RemoveContainer" containerID="49fffcc4c070312d631b0bae3974a38f95233aedbf72761b72cb4333bc249a47" Sep 10 13:51:25.107474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:25.107455 2559 scope.go:117] "RemoveContainer" containerID="62181f33d63b54f8d0ff08382542d209e0e07bc510216041905144537a476ce2" Sep 10 13:51:25.117293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:25.117272 2559 scope.go:117] "RemoveContainer" containerID="de7c0718e755c1610a5b7a3a646e8701ada900add0ff48c4b076800d0c6de8a9" Sep 10 13:51:42.722940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:42.722900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:51:47.690229 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:47.690184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:51:50.955284 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:51:50.955246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:07.589958 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:07.589919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:52:17.516586 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:17.516545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:17.619834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:17.619803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:24.500059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:24.500023 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:52:25.153118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:25.153085 2559 scope.go:117] "RemoveContainer" containerID="73a98f083dccc6ce8ae5d587a7d9d5e02a66020218a407cc54b2223afdfe8ce8" Sep 10 13:52:25.162737 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:25.162718 2559 scope.go:117] "RemoveContainer" containerID="13dba4b8ba63d3fa487147ee95201f12fb51868db9d51e9a2d61f66b39518974" Sep 10 13:52:26.841144 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:26.841106 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:28.649556 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:28.649513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:29.560356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:29.560317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:30.335380 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:30.335329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:31.421899 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:31.421872 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:32.354383 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:32.354335 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:33.190511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:33.190477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:38.282647 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:38.282611 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:52:43.385643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.385606 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:52:43.386081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.385865 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" podUID="b8f826da-514f-4220-9b3c-93d59e376d5e" containerName="maas-api" containerID="cri-o://f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f" gracePeriod=30 Sep 10 13:52:43.455679 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.455596 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:43.571242 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.571211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:43.665293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.665257 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:52:43.691538 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.691441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:43.738406 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.738380 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7x7\" (UniqueName: \"kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7\") pod \"b8f826da-514f-4220-9b3c-93d59e376d5e\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " Sep 10 13:52:43.738559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.738432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca\") pod \"b8f826da-514f-4220-9b3c-93d59e376d5e\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " Sep 10 13:52:43.738559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.738490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") pod \"b8f826da-514f-4220-9b3c-93d59e376d5e\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " Sep 10 13:52:43.738559 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.738543 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs\") pod \"b8f826da-514f-4220-9b3c-93d59e376d5e\" (UID: \"b8f826da-514f-4220-9b3c-93d59e376d5e\") " Sep 10 13:52:43.738801 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.738767 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8f826da-514f-4220-9b3c-93d59e376d5e" (UID: "b8f826da-514f-4220-9b3c-93d59e376d5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:52:43.740726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.740705 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8f826da-514f-4220-9b3c-93d59e376d5e" (UID: "b8f826da-514f-4220-9b3c-93d59e376d5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:52:43.740833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.740728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8f826da-514f-4220-9b3c-93d59e376d5e" (UID: "b8f826da-514f-4220-9b3c-93d59e376d5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:52:43.740833 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.740754 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7" (OuterVolumeSpecName: "kube-api-access-vc7x7") pod "b8f826da-514f-4220-9b3c-93d59e376d5e" (UID: "b8f826da-514f-4220-9b3c-93d59e376d5e"). InnerVolumeSpecName "kube-api-access-vc7x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:52:43.839473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.839439 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:52:43.839473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.839467 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f826da-514f-4220-9b3c-93d59e376d5e-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:52:43.839473 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.839477 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7x7\" (UniqueName: \"kubernetes.io/projected/b8f826da-514f-4220-9b3c-93d59e376d5e-kube-api-access-vc7x7\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:52:43.839709 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:43.839485 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f826da-514f-4220-9b3c-93d59e376d5e-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:52:44.111644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.111609 2559 generic.go:356] "Generic (PLEG): container finished" podID="b8f826da-514f-4220-9b3c-93d59e376d5e" containerID="f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f" exitCode=0 Sep 10 13:52:44.111827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.111662 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" Sep 10 13:52:44.111827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.111702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" event={"ID":"b8f826da-514f-4220-9b3c-93d59e376d5e","Type":"ContainerDied","Data":"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f"} Sep 10 13:52:44.111827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.111766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4" event={"ID":"b8f826da-514f-4220-9b3c-93d59e376d5e","Type":"ContainerDied","Data":"30f88f186701305e83097478c12ce28bd839769e03da34d167b62e51d699b541"} Sep 10 13:52:44.111827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.111797 2559 scope.go:117] "RemoveContainer" containerID="f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f" Sep 10 13:52:44.122297 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.122282 2559 scope.go:117] "RemoveContainer" containerID="f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f" Sep 10 13:52:44.122575 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:52:44.122559 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f\": container with ID starting with f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f not found: ID does not exist" containerID="f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f" Sep 10 13:52:44.122620 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.122584 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f"} err="failed to get container status \"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f\": rpc error: code = NotFound desc = could not find container \"f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f\": container with ID starting with f59cc308e1440ff5b69625478c546b9d76abfb8a6cb3755ebfa874edc631978f not found: ID does not exist" Sep 10 13:52:44.136955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.136935 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:52:44.141403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.141359 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a6883ccf-d7d6655fb-djzn4"] Sep 10 13:52:44.179481 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.179444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:44.919571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:44.919528 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f826da-514f-4220-9b3c-93d59e376d5e" path="/var/lib/kubelet/pods/b8f826da-514f-4220-9b3c-93d59e376d5e/volumes" Sep 10 13:52:45.077188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:45.077152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:46.157519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:46.157487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:47.117735 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:47.117689 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:47.279707 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:47.279677 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:48.734026 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:48.733976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:52.183060 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:52.183012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:52:54.101753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:52:54.101714 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:53:10.294452 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:10.294417 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:53:21.758572 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:21.758531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:27.092048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:27.092008 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:53:29.332433 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:29.332396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:30.040128 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:30.040088 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:36.401681 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:36.401642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:43.386927 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:43.386891 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:53:52.301236 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:52.301195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:52.401507 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:52.401473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:52.618535 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:52.618439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:53.864381 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:53.864338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:54.019998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:54.019964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:54.122107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:54.122026 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:54.700171 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:54.700136 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:54.800083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:54.800039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:54.901603 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:54.901574 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:58.238728 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:58.238687 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:53:59.389832 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:53:59.389792 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:54:03.044028 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:03.043979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:03.162518 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:03.162483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:03.738199 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:03.738168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:05.481989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:05.481949 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:54:05.520641 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:05.520603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:05.825475 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:05.825443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:06.831779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:06.831744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:06.945077 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:06.945049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:10.874037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:10.873991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:10.941944 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:10.941911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:11.013403 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:11.013355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:11.113232 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:11.113193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:16.986862 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:16.986829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:54:20.501654 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.501587 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" podUID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" containerName="main" containerID="cri-o://7483c2dc1b865f746e34108a4246e6ba112df557efc922bb4854965e0a0c50e7" gracePeriod=45 Sep 10 13:54:20.720921 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.720891 2559 generic.go:356] "Generic (PLEG): container finished" podID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" containerID="7483c2dc1b865f746e34108a4246e6ba112df557efc922bb4854965e0a0c50e7" exitCode=0 Sep 10 13:54:20.721082 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.720969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" event={"ID":"e393e8ec-aad8-41d8-9bc2-9533b4cec161","Type":"ContainerDied","Data":"7483c2dc1b865f746e34108a4246e6ba112df557efc922bb4854965e0a0c50e7"} Sep 10 13:54:20.759692 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.759671 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:54:20.794877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.794851 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.794884 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.794905 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.794953 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.794980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795226 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795037 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvnhb\" (UniqueName: \"kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb\") pod \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\" (UID: \"e393e8ec-aad8-41d8-9bc2-9533b4cec161\") " Sep 10 13:54:20.795284 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795224 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home" (OuterVolumeSpecName: "home") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:54:20.795341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795255 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache" (OuterVolumeSpecName: "model-cache") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:54:20.795341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:54:20.795457 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795390 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-home\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:20.795457 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.795411 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-model-cache\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:20.797322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.797298 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb" (OuterVolumeSpecName: "kube-api-access-jvnhb") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "kube-api-access-jvnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:54:20.797448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.797429 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm" (OuterVolumeSpecName: "dshm") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:54:20.797501 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.797414 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e393e8ec-aad8-41d8-9bc2-9533b4cec161" (UID: "e393e8ec-aad8-41d8-9bc2-9533b4cec161"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:54:20.895923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.895878 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnhb\" (UniqueName: \"kubernetes.io/projected/e393e8ec-aad8-41d8-9bc2-9533b4cec161-kube-api-access-jvnhb\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:20.895923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.895920 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-dshm\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:20.895923 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.895934 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tls-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:20.896142 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:20.895968 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e393e8ec-aad8-41d8-9bc2-9533b4cec161-tmp-dir\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:54:21.597940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.597908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:21.727747 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.727720 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" Sep 10 13:54:21.727913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.727752 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x" event={"ID":"e393e8ec-aad8-41d8-9bc2-9533b4cec161","Type":"ContainerDied","Data":"cd55f5e1700452e50f33bcdc9389c63e7670c2a7ce8fdd8748591c699469e67c"} Sep 10 13:54:21.727913 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.727803 2559 scope.go:117] "RemoveContainer" containerID="7483c2dc1b865f746e34108a4246e6ba112df557efc922bb4854965e0a0c50e7" Sep 10 13:54:21.752852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.752825 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:54:21.760755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.760734 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/auth-test-model-579ae1-kserve-cf6794bff-z5h5x"] Sep 10 13:54:21.829431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:21.829400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:22.919056 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:22.919014 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" path="/var/lib/kubelet/pods/e393e8ec-aad8-41d8-9bc2-9533b4cec161/volumes" Sep 10 13:54:25.212006 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:25.211977 2559 scope.go:117] "RemoveContainer" containerID="88a8a6a8fa11e240c3fbc9d6557a43dc5f437e62c29a2efe99c90bc0d59f2411" Sep 10 13:54:30.793104 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:30.793060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:35.182951 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:35.182914 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:54:42.677448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:42.677407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:42.828724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:42.828690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:43.487078 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:43.487043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:49.728541 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:49.728502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:49.810919 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:49.810887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:50.120209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.120182 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:50.438996 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.438921 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:54:50.439430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439415 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f826da-514f-4220-9b3c-93d59e376d5e" containerName="maas-api" Sep 10 13:54:50.439430 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439431 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f826da-514f-4220-9b3c-93d59e376d5e" containerName="maas-api" Sep 10 13:54:50.439536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439442 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" containerName="main" Sep 10 13:54:50.439536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439451 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" containerName="main" Sep 10 13:54:50.439536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439521 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e393e8ec-aad8-41d8-9bc2-9533b4cec161" containerName="main" Sep 10 13:54:50.439536 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.439533 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f826da-514f-4220-9b3c-93d59e376d5e" containerName="maas-api" Sep 10 13:54:50.442866 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.442840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.445462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.445441 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d5fa55-openshift-default-dockercfg-l7jhk\"" Sep 10 13:54:50.460441 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.460421 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:54:50.496262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.496240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:50.544694 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.544650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.544994 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.544970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.545201 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.545184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.545389 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.545356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqksj\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.545591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.545576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.545744 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.545730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.545918 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.545884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.546110 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.546094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.546289 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.546273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647342 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647506 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647566 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqksj\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647680 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647863 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.647938 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.647908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.648249 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.648213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.648385 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.648274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.648385 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.648290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.648385 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.648303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.649814 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.649793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.650127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.650107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.662327 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.662307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqksj\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.678218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.678196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token\") pod \"e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:50.753561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:50.753507 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:51.038323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.038295 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:54:51.040196 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:54:51.040171 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc886f00f_23b6_4683_8ad2_3dda7e6346bd.slice/crio-c8725b50ba864028ae36acb192182f6fdea97ee93c4b873db59b90875d15778f WatchSource:0}: Error finding container c8725b50ba864028ae36acb192182f6fdea97ee93c4b873db59b90875d15778f: Status 404 returned error can't find the container with id c8725b50ba864028ae36acb192182f6fdea97ee93c4b873db59b90875d15778f Sep 10 13:54:51.042184 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.042148 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:54:51.042288 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.042227 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:54:51.042288 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.042270 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:54:51.121127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.121099 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:51.922984 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.922945 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerStarted","Data":"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b"} Sep 10 13:54:51.923398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.923015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerStarted","Data":"c8725b50ba864028ae36acb192182f6fdea97ee93c4b873db59b90875d15778f"} Sep 10 13:54:51.962643 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:51.962595 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podStartSLOduration=1.962576897 podStartE2EDuration="1.962576897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:54:51.961848363 +0000 UTC m=+1467.571633275" watchObservedRunningTime="2026-09-10 13:54:51.962576897 +0000 UTC m=+1467.572361809" Sep 10 13:54:52.754092 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:52.754057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:52.759196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:52.759172 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:52.929730 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:52.929704 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:52.930472 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:52.930455 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:54:53.360355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:53.360310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:53.973001 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:53.972962 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:54.073046 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:54.073019 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:54.418846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:54.418817 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:55.799619 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:55.799584 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:56.866434 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:56.866393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:57.722846 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:57.722809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:58.810497 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.810462 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:54:58.822055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.822017 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:58.824486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.824464 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d5fa55-serving-cert\"" Sep 10 13:54:58.835291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.835267 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:54:58.872395 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.872355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:58.924539 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.924515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:58.924648 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.924559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:58.924648 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.924620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:58.924648 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.924637 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkc4n\" (UniqueName: \"kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:58.969105 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:58.969071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:59.025185 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.025155 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.025325 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.025201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.025325 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.025243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.025325 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.025265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkc4n\" (UniqueName: \"kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.025547 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:54:59.025416 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-serving-cert: secret "maas-api-e2e-api-a-d5fa55-serving-cert" not found Sep 10 13:54:59.025547 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:54:59.025478 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls podName:55153c8a-3665-49af-9563-b3e5e16eb7e6 nodeName:}" failed. No retries permitted until 2026-09-10 13:54:59.525461757 +0000 UTC m=+1475.135246661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls") pod "maas-api-e2e-api-a-d5fa55-894486647-rksk7" (UID: "55153c8a-3665-49af-9563-b3e5e16eb7e6") : secret "maas-api-e2e-api-a-d5fa55-serving-cert" not found Sep 10 13:54:59.025979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.025956 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.027716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.027693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.039057 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.039036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkc4n\" (UniqueName: \"kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.398890 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.398853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:59.499571 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.499536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:59.530123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.530098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.532387 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.532347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") pod \"maas-api-e2e-api-a-d5fa55-894486647-rksk7\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:54:59.578095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.577999 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:54:59.735521 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:54:59.735430 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:55:00.010532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.010507 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:55:00.013127 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:55:00.013100 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55153c8a_3665_49af_9563_b3e5e16eb7e6.slice/crio-c62edbbfe963516f85cb79034fb6eed06806e815d93d024ccbfd77a95377ffb3 WatchSource:0}: Error finding container c62edbbfe963516f85cb79034fb6eed06806e815d93d024ccbfd77a95377ffb3: Status 404 returned error can't find the container with id c62edbbfe963516f85cb79034fb6eed06806e815d93d024ccbfd77a95377ffb3 Sep 10 13:55:00.014451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.014432 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:55:00.979013 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.978938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" event={"ID":"55153c8a-3665-49af-9563-b3e5e16eb7e6","Type":"ContainerStarted","Data":"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a"} Sep 10 13:55:00.979013 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.978990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" event={"ID":"55153c8a-3665-49af-9563-b3e5e16eb7e6","Type":"ContainerStarted","Data":"c62edbbfe963516f85cb79034fb6eed06806e815d93d024ccbfd77a95377ffb3"} Sep 10 13:55:00.979205 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.979077 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:55:00.999414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:00.999335 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" podStartSLOduration=2.380222387 podStartE2EDuration="2.999322149s" totalImagesPullingTime="619.099762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:00.999160515 +0000 UTC m=+1476.608945429" watchObservedRunningTime="2026-09-10 13:55:00.999322149 +0000 UTC m=+1476.609107068" Sep 10 13:55:01.248224 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:01.248107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:04.286627 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:04.286588 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:55:06.889350 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:06.889293 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:06.992283 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:06.992251 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:55:08.021584 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:08.021545 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:14.068504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.068437 2559 generic.go:356] "Generic (PLEG): container finished" podID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerID="4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b" exitCode=0 Sep 10 13:55:14.068504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.068498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerDied","Data":"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b"} Sep 10 13:55:14.068924 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.068807 2559 scope.go:117] "RemoveContainer" containerID="4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b" Sep 10 13:55:14.069891 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.069863 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:55:14.069961 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.069932 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:55:14.070003 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:14.069961 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 13:55:15.076607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:15.076571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerStarted","Data":"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f"} Sep 10 13:55:15.754150 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:15.754110 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:55:15.754353 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:15.754326 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:15.754453 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:15.754402 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:16.754704 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:16.754669 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:16.755074 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:16.754721 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:17.754705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:17.754666 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:17.755108 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:17.754733 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:18.754796 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:18.754757 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:18.755171 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:18.754813 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:19.754335 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:19.754297 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:19.754564 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:19.754355 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:19.857962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:19.857929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:20.754103 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:20.754071 2559 patch_prober.go:28] interesting pod/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 10 13:55:20.754285 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:20.754122 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 10 13:55:21.559150 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:21.559113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:21.757857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:21.757828 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:55:21.758409 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:21.758361 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:55:21.759015 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:21.758998 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:55:22.586511 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:22.586477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:23.143543 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:23.143512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:55:23.698211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:23.698169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:24.095594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:24.095562 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:24.595550 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:24.595513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:24.882728 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:24.882647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:25.025456 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.025406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:25.048960 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.048932 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:55:25.052829 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.052802 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 13:55:25.128421 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.128356 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:25.238084 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.238011 2559 scope.go:117] "RemoveContainer" containerID="fba984d2d962f579026fb75242ab92b562d9b230748c7dd936a9fdd25069f88d" Sep 10 13:55:25.247323 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:25.247303 2559 scope.go:117] "RemoveContainer" containerID="4052b7e257abc8d8b0ac43bec3fd5e1ef429fcd2234f808958ab84a1e1452d21" Sep 10 13:55:26.061314 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:26.061275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:27.132405 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:27.132351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:28.132897 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:28.132865 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:29.673207 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:29.673176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:30.339876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:30.339847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:31.517658 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:31.517626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:32.473286 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:32.473254 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:33.653687 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:33.653645 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:34.612901 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.612868 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:55:34.618298 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.618276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.620838 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.620817 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fc1974-serving-cert\"" Sep 10 13:55:34.632907 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.632886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:55:34.655975 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.655950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:34.744411 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.744354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.744602 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.744517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrzx\" (UniqueName: \"kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.744602 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.744576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.744726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.744672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.745164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.745136 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:34.845442 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.845410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.845442 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.845447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.845640 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.845506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrzx\" (UniqueName: \"kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.845640 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.845546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.846513 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.846487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.848216 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.848196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.848341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.848258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.858779 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.858757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrzx\" (UniqueName: \"kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx\") pod \"maas-api-e2e-api-b-fc1974-6469b75884-cp5gb\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:34.928129 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:34.928071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:35.217796 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:35.217770 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:55:35.218465 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:55:35.218441 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2c9501_3fb1_4b54_a69a_b711eacaca27.slice/crio-ddb2aee2bc651869e73075d754fe3c2aabb95de3ec236303a61ae97c4dd42a7c WatchSource:0}: Error finding container ddb2aee2bc651869e73075d754fe3c2aabb95de3ec236303a61ae97c4dd42a7c: Status 404 returned error can't find the container with id ddb2aee2bc651869e73075d754fe3c2aabb95de3ec236303a61ae97c4dd42a7c Sep 10 13:55:35.364318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:35.364282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:35.464166 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:35.464132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:36.014260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:36.014221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:36.202018 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:36.201985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" event={"ID":"0f2c9501-3fb1-4b54-a69a-b711eacaca27","Type":"ContainerStarted","Data":"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c"} Sep 10 13:55:36.202195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:36.202045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" event={"ID":"0f2c9501-3fb1-4b54-a69a-b711eacaca27","Type":"ContainerStarted","Data":"ddb2aee2bc651869e73075d754fe3c2aabb95de3ec236303a61ae97c4dd42a7c"} Sep 10 13:55:36.202195 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:36.202085 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:36.236048 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:36.235997 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" podStartSLOduration=1.905889865 podStartE2EDuration="2.235982288s" totalImagesPullingTime="330.092423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:36.230683273 +0000 UTC m=+1511.840468185" watchObservedRunningTime="2026-09-10 13:55:36.235982288 +0000 UTC m=+1511.845767201" Sep 10 13:55:37.257665 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:37.257620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:42.214382 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:42.214338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:55:42.900644 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:42.900610 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:55:48.965125 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:48.965039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:55.972876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:55.972833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:58.274851 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:58.274809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:58.568998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:58.568913 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:55:59.753924 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:55:59.753889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:00.807162 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:00.807133 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:01.818072 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:01.818029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:02.911515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:02.911475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:03.894421 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:03.894355 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:05.084179 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:05.084146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:06.203203 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:06.203163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:06.298425 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:06.298392 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:56:06.919187 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:06.919154 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:07.580982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:07.580950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:09.999076 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:09.999042 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:10.109906 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:10.109868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:11.702391 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:11.702331 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.480877 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.480841 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:56:17.481329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.481139 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" podUID="55153c8a-3665-49af-9563-b3e5e16eb7e6" containerName="maas-api" containerID="cri-o://d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a" gracePeriod=30 Sep 10 13:56:17.547312 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.547278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.642291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.642251 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.734087 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.734015 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:56:17.743379 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.743344 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.823879 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.823849 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkc4n\" (UniqueName: \"kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n\") pod \"55153c8a-3665-49af-9563-b3e5e16eb7e6\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " Sep 10 13:56:17.824039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.823917 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca\") pod \"55153c8a-3665-49af-9563-b3e5e16eb7e6\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " Sep 10 13:56:17.824039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.823965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") pod \"55153c8a-3665-49af-9563-b3e5e16eb7e6\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " Sep 10 13:56:17.824039 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.824028 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs\") pod \"55153c8a-3665-49af-9563-b3e5e16eb7e6\" (UID: \"55153c8a-3665-49af-9563-b3e5e16eb7e6\") " Sep 10 13:56:17.824287 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.824261 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55153c8a-3665-49af-9563-b3e5e16eb7e6" (UID: "55153c8a-3665-49af-9563-b3e5e16eb7e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:56:17.824405 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.824361 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55153c8a-3665-49af-9563-b3e5e16eb7e6-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:17.826045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.826002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55153c8a-3665-49af-9563-b3e5e16eb7e6" (UID: "55153c8a-3665-49af-9563-b3e5e16eb7e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:56:17.826045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.826029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55153c8a-3665-49af-9563-b3e5e16eb7e6" (UID: "55153c8a-3665-49af-9563-b3e5e16eb7e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:56:17.826204 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.826085 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n" (OuterVolumeSpecName: "kube-api-access-xkc4n") pod "55153c8a-3665-49af-9563-b3e5e16eb7e6" (UID: "55153c8a-3665-49af-9563-b3e5e16eb7e6"). InnerVolumeSpecName "kube-api-access-xkc4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:56:17.842389 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.842323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.923546 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.923510 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:17.925827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.925805 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkc4n\" (UniqueName: \"kubernetes.io/projected/55153c8a-3665-49af-9563-b3e5e16eb7e6-kube-api-access-xkc4n\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:17.925827 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.925829 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:17.925982 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:17.925838 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55153c8a-3665-49af-9563-b3e5e16eb7e6-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:18.186178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.186147 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:18.336107 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.336075 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:18.467893 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.467814 2559 generic.go:356] "Generic (PLEG): container finished" podID="55153c8a-3665-49af-9563-b3e5e16eb7e6" containerID="d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a" exitCode=0 Sep 10 13:56:18.467893 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.467861 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" Sep 10 13:56:18.468083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.467903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" event={"ID":"55153c8a-3665-49af-9563-b3e5e16eb7e6","Type":"ContainerDied","Data":"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a"} Sep 10 13:56:18.468083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.467955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7" event={"ID":"55153c8a-3665-49af-9563-b3e5e16eb7e6","Type":"ContainerDied","Data":"c62edbbfe963516f85cb79034fb6eed06806e815d93d024ccbfd77a95377ffb3"} Sep 10 13:56:18.468083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.467978 2559 scope.go:117] "RemoveContainer" containerID="d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a" Sep 10 13:56:18.478177 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.478158 2559 scope.go:117] "RemoveContainer" containerID="d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a" Sep 10 13:56:18.478456 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:56:18.478437 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a\": container with ID starting with d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a not found: ID does not exist" containerID="d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a" Sep 10 13:56:18.478519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.478464 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a"} err="failed to get container status \"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a\": rpc error: code = NotFound desc = could not find container \"d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a\": container with ID starting with d70b3df12440aa8322e8061e5393de05e14cbf02c6bb2dbf9147475695b5073a not found: ID does not exist" Sep 10 13:56:18.497931 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.497901 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:56:18.500646 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.500627 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5fa55-894486647-rksk7"] Sep 10 13:56:18.919500 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:18.919468 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55153c8a-3665-49af-9563-b3e5e16eb7e6" path="/var/lib/kubelet/pods/55153c8a-3665-49af-9563-b3e5e16eb7e6/volumes" Sep 10 13:56:19.224073 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:19.223972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:20.215347 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:20.215315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:21.220017 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.219985 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:21.428455 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.428421 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:56:21.628034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.628003 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:56:21.628482 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.628469 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55153c8a-3665-49af-9563-b3e5e16eb7e6" containerName="maas-api" Sep 10 13:56:21.628534 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.628484 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="55153c8a-3665-49af-9563-b3e5e16eb7e6" containerName="maas-api" Sep 10 13:56:21.628573 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.628556 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="55153c8a-3665-49af-9563-b3e5e16eb7e6" containerName="maas-api" Sep 10 13:56:21.632980 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.632961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.644800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.644778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"kube-root-ca.crt\"" Sep 10 13:56:21.644800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.644791 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"default-dockercfg-rjdk6\"" Sep 10 13:56:21.645019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.644779 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"openshift-service-ca.crt\"" Sep 10 13:56:21.645019 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.644779 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-579ae1\"/\"rate-test-model-579ae1-kserve-self-signed-certs\"" Sep 10 13:56:21.684270 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.684246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:56:21.733803 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.733764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:21.759029 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwj7\" (UniqueName: \"kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.759154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759037 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.759154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.759154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.759318 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.759409 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.759314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.860879 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.860836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.860929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwj7\" (UniqueName: \"kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.860962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.861020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.861052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.861078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.861607 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.861580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.862173 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.862138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.862293 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.862262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.864865 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.864815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.865110 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.865042 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.880181 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.880126 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwj7\" (UniqueName: \"kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7\") pod \"rate-test-model-579ae1-kserve-8678577d99-cbcp7\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:21.942855 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:21.942827 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:22.219034 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:22.219008 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:56:22.220389 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:56:22.220352 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf47a33ff_ed9c_40d5_9a15_20b4bf28fac6.slice/crio-f31608d47b67a350ca0e4e6463c1c6842cadfa72518647d6c47a6b83236fcd5d WatchSource:0}: Error finding container f31608d47b67a350ca0e4e6463c1c6842cadfa72518647d6c47a6b83236fcd5d: Status 404 returned error can't find the container with id f31608d47b67a350ca0e4e6463c1c6842cadfa72518647d6c47a6b83236fcd5d Sep 10 13:56:22.455551 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:22.455505 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:22.495454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:22.495346 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" event={"ID":"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6","Type":"ContainerStarted","Data":"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e"} Sep 10 13:56:22.495454 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:22.495423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" event={"ID":"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6","Type":"ContainerStarted","Data":"f31608d47b67a350ca0e4e6463c1c6842cadfa72518647d6c47a6b83236fcd5d"} Sep 10 13:56:22.521818 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:22.521766 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" podStartSLOduration=1.521750921 podStartE2EDuration="1.521750921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:22.521277787 +0000 UTC m=+1558.131062701" watchObservedRunningTime="2026-09-10 13:56:22.521750921 +0000 UTC m=+1558.131535834" Sep 10 13:56:23.857500 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:23.857444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:26.156952 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:26.156911 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:26.256223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:26.256193 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:26.501457 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:26.501379 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:56:26.501702 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:26.501672 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" containerID="cri-o://248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f" gracePeriod=30 Sep 10 13:56:26.543730 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:26.543693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:27.233103 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:27.233067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:28.207225 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:28.207186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:29.244361 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:29.244319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:30.110738 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:30.110703 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:31.258834 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.258791 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:31.755110 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.755086 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:56:31.852356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852283 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852314 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852356 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852412 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqksj\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852613 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852448 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852668 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852722 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852597 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:56:31.852885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852746 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:56:31.852885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.852885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852828 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:56:31.852885 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852852 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy\") pod \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\" (UID: \"c886f00f-23b6-4683-8ad2-3dda7e6346bd\") " Sep 10 13:56:31.853081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.852956 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:56:31.853163 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:56:31.853292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853275 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.853341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853297 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.853341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853313 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.853341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853325 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.853341 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.853338 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.854749 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.854720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:56:31.854876 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.854857 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:56:31.855009 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.854991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj" (OuterVolumeSpecName: "kube-api-access-sqksj") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "kube-api-access-sqksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:56:31.855059 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.855013 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "c886f00f-23b6-4683-8ad2-3dda7e6346bd" (UID: "c886f00f-23b6-4683-8ad2-3dda7e6346bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:56:31.943462 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.943436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:31.943582 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.943474 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:31.954420 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.954399 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.954508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.954422 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.954508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.954435 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqksj\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-kube-api-access-sqksj\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.954508 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.954452 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c886f00f-23b6-4683-8ad2-3dda7e6346bd-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:31.955962 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:31.955933 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:32.395726 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.395690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:32.556515 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.556477 2559 generic.go:356] "Generic (PLEG): container finished" podID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerID="248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f" exitCode=0 Sep 10 13:56:32.556673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.556544 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" Sep 10 13:56:32.556673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.556569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerDied","Data":"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f"} Sep 10 13:56:32.556673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.556644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds" event={"ID":"c886f00f-23b6-4683-8ad2-3dda7e6346bd","Type":"ContainerDied","Data":"c8725b50ba864028ae36acb192182f6fdea97ee93c4b873db59b90875d15778f"} Sep 10 13:56:32.556673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.556668 2559 scope.go:117] "RemoveContainer" containerID="248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f" Sep 10 13:56:32.568151 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.568136 2559 scope.go:117] "RemoveContainer" containerID="4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b" Sep 10 13:56:32.570662 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.570639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:56:32.576615 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.576597 2559 scope.go:117] "RemoveContainer" containerID="248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f" Sep 10 13:56:32.576882 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:56:32.576863 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f\": container with ID starting with 248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f not found: ID does not exist" containerID="248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f" Sep 10 13:56:32.576930 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.576890 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f"} err="failed to get container status \"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f\": rpc error: code = NotFound desc = could not find container \"248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f\": container with ID starting with 248e91df9f2683e16a721c17d9ea9f1369f56ec8dd91a477bfcf4f0c70a5057f not found: ID does not exist" Sep 10 13:56:32.576930 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.576905 2559 scope.go:117] "RemoveContainer" containerID="4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b" Sep 10 13:56:32.577165 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:56:32.577146 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b\": container with ID starting with 4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b not found: ID does not exist" containerID="4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b" Sep 10 13:56:32.577209 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.577170 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b"} err="failed to get container status \"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b\": rpc error: code = NotFound desc = could not find container \"4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b\": container with ID starting with 4297e33632e90456e6772a493a1aea5c365348f4073903a8d5b61d7a5e082b6b not found: ID does not exist" Sep 10 13:56:32.588790 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.588770 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:56:32.601143 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.601122 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d5fa55-openshift-default-54dd6456fb-gq4ds"] Sep 10 13:56:32.919113 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:32.919081 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" path="/var/lib/kubelet/pods/c886f00f-23b6-4683-8ad2-3dda7e6346bd/volumes" Sep 10 13:56:33.435354 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:33.435314 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:36.852062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:36.852031 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:56:42.661410 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.661377 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:56:42.661890 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.661576 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" podUID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" containerName="maas-api" containerID="cri-o://d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c" gracePeriod=30 Sep 10 13:56:42.696321 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.696294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:42.815287 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.815256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:42.912963 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.912889 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:42.922981 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:42.922959 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:56:43.004044 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.004012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:43.047968 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.047941 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs\") pod \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " Sep 10 13:56:43.048094 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.048000 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca\") pod \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " Sep 10 13:56:43.048094 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.048017 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls\") pod \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " Sep 10 13:56:43.048094 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.048046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcrzx\" (UniqueName: \"kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx\") pod \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\" (UID: \"0f2c9501-3fb1-4b54-a69a-b711eacaca27\") " Sep 10 13:56:43.048448 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.048421 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f2c9501-3fb1-4b54-a69a-b711eacaca27" (UID: "0f2c9501-3fb1-4b54-a69a-b711eacaca27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:56:43.050123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.050092 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f2c9501-3fb1-4b54-a69a-b711eacaca27" (UID: "0f2c9501-3fb1-4b54-a69a-b711eacaca27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:56:43.050123 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.050098 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx" (OuterVolumeSpecName: "kube-api-access-rcrzx") pod "0f2c9501-3fb1-4b54-a69a-b711eacaca27" (UID: "0f2c9501-3fb1-4b54-a69a-b711eacaca27"). InnerVolumeSpecName "kube-api-access-rcrzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:56:43.050264 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.050141 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f2c9501-3fb1-4b54-a69a-b711eacaca27" (UID: "0f2c9501-3fb1-4b54-a69a-b711eacaca27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:56:43.149734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.149700 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:43.149734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.149727 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2c9501-3fb1-4b54-a69a-b711eacaca27-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:43.149734 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.149739 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f2c9501-3fb1-4b54-a69a-b711eacaca27-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:43.149998 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.149747 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrzx\" (UniqueName: \"kubernetes.io/projected/0f2c9501-3fb1-4b54-a69a-b711eacaca27-kube-api-access-rcrzx\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:56:43.339444 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.339393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:43.451188 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.451141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:43.628045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.627954 2559 generic.go:356] "Generic (PLEG): container finished" podID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" containerID="d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c" exitCode=0 Sep 10 13:56:43.628045 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.628037 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" event={"ID":"0f2c9501-3fb1-4b54-a69a-b711eacaca27","Type":"ContainerDied","Data":"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c"} Sep 10 13:56:43.628273 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.628085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" event={"ID":"0f2c9501-3fb1-4b54-a69a-b711eacaca27","Type":"ContainerDied","Data":"ddb2aee2bc651869e73075d754fe3c2aabb95de3ec236303a61ae97c4dd42a7c"} Sep 10 13:56:43.628273 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.628113 2559 scope.go:117] "RemoveContainer" containerID="d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c" Sep 10 13:56:43.628398 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.628280 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb" Sep 10 13:56:43.638037 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.638009 2559 scope.go:117] "RemoveContainer" containerID="d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c" Sep 10 13:56:43.638302 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:56:43.638283 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c\": container with ID starting with d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c not found: ID does not exist" containerID="d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c" Sep 10 13:56:43.638344 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.638310 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c"} err="failed to get container status \"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c\": rpc error: code = NotFound desc = could not find container \"d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c\": container with ID starting with d3427fcfaf61346b3a93cf61770f8efc443f22b7454e0ad615529aec6397938c not found: ID does not exist" Sep 10 13:56:43.652160 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.652136 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:56:43.656845 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:43.656824 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc1974-6469b75884-cp5gb"] Sep 10 13:56:44.475288 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:44.475257 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:44.918657 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:44.918625 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" path="/var/lib/kubelet/pods/0f2c9501-3fb1-4b54-a69a-b711eacaca27/volumes" Sep 10 13:56:45.444355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:45.444320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:46.382979 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:46.382946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:47.487423 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:47.487388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:49.984302 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:49.984262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:56:50.141719 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:50.141686 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:50.208889 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:50.208858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:50.446439 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:50.446406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:51.729989 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:51.729954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:52.731176 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:52.731141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:53.646466 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:53.646428 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:54.619116 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:54.619079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:56:55.815917 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:56:55.814963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:01.471384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.471338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:01.567891 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.567846 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:57:01.568081 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.568055 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" podUID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" containerName="payload-pre-processing" containerID="cri-o://e485ccffc3ed3ed6eccac3afbd5c8f8f630b7d52c683375a0d2b40e218d28fc2" gracePeriod=30 Sep 10 13:57:01.605755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.605719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:01.706642 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.706569 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:01.748392 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.748310 2559 generic.go:356] "Generic (PLEG): container finished" podID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" containerID="e485ccffc3ed3ed6eccac3afbd5c8f8f630b7d52c683375a0d2b40e218d28fc2" exitCode=0 Sep 10 13:57:01.748527 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.748405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" event={"ID":"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd","Type":"ContainerDied","Data":"e485ccffc3ed3ed6eccac3afbd5c8f8f630b7d52c683375a0d2b40e218d28fc2"} Sep 10 13:57:01.828178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.828154 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:57:01.875954 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.875898 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:01.905437 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.905404 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thbpc\" (UniqueName: \"kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc\") pod \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " Sep 10 13:57:01.905600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.905455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume\") pod \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\" (UID: \"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd\") " Sep 10 13:57:01.905813 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.905790 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" (UID: "5ff571aa-bbf8-4010-8bd0-e0001e9a34fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:01.905882 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.905864 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-plugins-config-volume\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:01.907605 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.907583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc" (OuterVolumeSpecName: "kube-api-access-thbpc") pod "5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" (UID: "5ff571aa-bbf8-4010-8bd0-e0001e9a34fd"). InnerVolumeSpecName "kube-api-access-thbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:01.949688 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:01.949658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:02.007220 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.007194 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbpc\" (UniqueName: \"kubernetes.io/projected/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd-kube-api-access-thbpc\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:02.343329 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.343250 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:02.480062 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.480012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:02.624486 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.624393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:02.755973 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.755949 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" Sep 10 13:57:02.756118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.755971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4" event={"ID":"5ff571aa-bbf8-4010-8bd0-e0001e9a34fd","Type":"ContainerDied","Data":"a8b61efd7d9975e5042e7f1ed3a8328667a5b11bb045f670fd4d84f187e50d3e"} Sep 10 13:57:02.756118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.756025 2559 scope.go:117] "RemoveContainer" containerID="e485ccffc3ed3ed6eccac3afbd5c8f8f630b7d52c683375a0d2b40e218d28fc2" Sep 10 13:57:02.797291 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.797266 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:57:02.817519 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.817484 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3d0caa-f76b667c4-58fs4"] Sep 10 13:57:02.925292 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:02.925231 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" path="/var/lib/kubelet/pods/5ff571aa-bbf8-4010-8bd0-e0001e9a34fd/volumes" Sep 10 13:57:03.682504 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:03.682469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:03.775303 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:03.775267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:04.867523 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:04.867490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:05.823422 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:05.823385 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:07.703451 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:07.703409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:57:09.842309 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:09.842276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:09.909764 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:09.909710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:10.115218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:10.115134 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:10.895673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:10.895642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:12.021384 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:12.021341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:12.827786 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:12.827746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:13.834724 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:13.834671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:14.728118 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:14.728079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:21.468400 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:21.468298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:21.619514 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:21.619477 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:21.714783 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:21.714735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:24.268568 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:24.268532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:24.368739 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:24.368701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:24.640845 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:24.640755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:26.699570 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:26.699533 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:57:29.248621 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:29.248585 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:29.310262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:29.310229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:29.547429 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:29.547399 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:57:29.547609 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:29.547588 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" containerID="cri-o://396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e" gracePeriod=30 Sep 10 13:57:29.551117 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:29.551084 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:34.792568 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.792545 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:57:34.900346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900275 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d9zr\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900346 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900388 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900442 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900468 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900504 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900553 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900519 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.900872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900794 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:57:34.900872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:57:34.900872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900843 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 13:57:34.901043 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900925 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.901043 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.900957 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs\") pod \"bcf587d2-c502-480c-b40f-7bc25c60824e\" (UID: \"bcf587d2-c502-480c-b40f-7bc25c60824e\") " Sep 10 13:57:34.901175 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901148 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:57:34.901414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901394 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:34.901414 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901416 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf587d2-c502-480c-b40f-7bc25c60824e-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:34.901577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901413 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:57:34.901577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901429 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:34.901577 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.901441 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:34.902743 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.902720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr" (OuterVolumeSpecName: "kube-api-access-5d9zr") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "kube-api-access-5d9zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:34.902825 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.902737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 13:57:34.903008 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.902979 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:57:34.903108 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.903039 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcf587d2-c502-480c-b40f-7bc25c60824e" (UID: "bcf587d2-c502-480c-b40f-7bc25c60824e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:57:34.962127 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.962099 2559 generic.go:356] "Generic (PLEG): container finished" podID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerID="396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e" exitCode=0 Sep 10 13:57:34.962230 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.962196 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" Sep 10 13:57:34.962269 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.962230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerDied","Data":"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e"} Sep 10 13:57:34.962345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.962286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g" event={"ID":"bcf587d2-c502-480c-b40f-7bc25c60824e","Type":"ContainerDied","Data":"0d799a8c07e38da23420af55b1a6bed3fc4378d546a35364b1f5f1ec4c157899"} Sep 10 13:57:34.962345 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.962308 2559 scope.go:117] "RemoveContainer" containerID="396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e" Sep 10 13:57:34.971608 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.971588 2559 scope.go:117] "RemoveContainer" containerID="17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b" Sep 10 13:57:34.979826 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.979810 2559 scope.go:117] "RemoveContainer" containerID="396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e" Sep 10 13:57:34.980096 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:57:34.980080 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e\": container with ID starting with 396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e not found: ID does not exist" containerID="396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e" Sep 10 13:57:34.980140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.980103 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e"} err="failed to get container status \"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e\": rpc error: code = NotFound desc = could not find container \"396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e\": container with ID starting with 396324b2a9f2e7bee18c7d4a1ff8ec612e1d5948e137c9c45e3bfcb28b2fc52e not found: ID does not exist" Sep 10 13:57:34.980140 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.980119 2559 scope.go:117] "RemoveContainer" containerID="17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b" Sep 10 13:57:34.980358 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:57:34.980341 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b\": container with ID starting with 17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b not found: ID does not exist" containerID="17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b" Sep 10 13:57:34.980441 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.980361 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b"} err="failed to get container status \"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b\": rpc error: code = NotFound desc = could not find container \"17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b\": container with ID starting with 17350986ad3de5e85aaf2ef5d80cc1e5d5085474acba1f2d0aed7d8d47c8855b not found: ID does not exist" Sep 10 13:57:34.995590 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:34.995570 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:57:35.000322 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.000303 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b3723e-openshift-default-79b5456544-5km6g"] Sep 10 13:57:35.002005 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.001981 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9zr\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-kube-api-access-5d9zr\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:35.002005 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.002003 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:35.002154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.002013 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:35.002154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.002022 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:35.002154 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:35.002041 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf587d2-c502-480c-b40f-7bc25c60824e-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:57:36.919083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:36.919049 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" path="/var/lib/kubelet/pods/bcf587d2-c502-480c-b40f-7bc25c60824e/volumes" Sep 10 13:57:40.406727 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:40.406693 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:57:52.596282 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:52.596251 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:57:56.276753 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:56.276713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:56.386509 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:56.386480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:56.786760 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:56.786727 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:57:56.850520 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:56.850488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:57.908461 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:57.908432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:58.371337 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:58.371299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:58.722349 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:58.722270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:57:58.979254 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:57:58.979162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:01.671359 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:01.671324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:02.341781 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:02.341738 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:02.434424 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:02.434390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:02.536007 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:02.535977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:02.777838 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:02.777790 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:03.854950 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:03.854915 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:04.804211 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804170 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 13:58:04.804800 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804780 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.804857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804805 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.804857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804828 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.804857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804838 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.804857 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804851 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804861 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804874 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804882 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804916 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" containerName="maas-api" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804924 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" containerName="maas-api" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804936 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" containerName="payload-pre-processing" Sep 10 13:58:04.804985 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.804945 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" containerName="payload-pre-processing" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805049 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805061 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805071 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c886f00f-23b6-4683-8ad2-3dda7e6346bd" containerName="istio-proxy" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805085 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf587d2-c502-480c-b40f-7bc25c60824e" containerName="istio-proxy" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805093 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2c9501-3fb1-4b54-a69a-b711eacaca27" containerName="maas-api" Sep 10 13:58:04.805196 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.805104 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff571aa-bbf8-4010-8bd0-e0001e9a34fd" containerName="payload-pre-processing" Sep 10 13:58:04.808705 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.808685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.812129 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.812107 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0fe383-serving-cert\"" Sep 10 13:58:04.819532 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.819508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 13:58:04.842054 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.842019 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:04.851240 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.851214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.851387 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.851343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.851485 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.851415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.851548 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.851518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqztg\" (UniqueName: \"kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.952561 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.952525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.953024 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.952574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.953024 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.952609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.953024 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:58:04.952695 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-serving-cert: secret "maas-api-e2e-ipp-a-0fe383-serving-cert" not found Sep 10 13:58:04.953024 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:58:04.952765 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls podName:7a97a0d5-dba4-4957-9219-00645ba985fe nodeName:}" failed. No retries permitted until 2026-09-10 13:58:05.452743016 +0000 UTC m=+1661.062528126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls") pod "maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" (UID: "7a97a0d5-dba4-4957-9219-00645ba985fe") : secret "maas-api-e2e-ipp-a-0fe383-serving-cert" not found Sep 10 13:58:04.953024 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.952790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqztg\" (UniqueName: \"kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.953426 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.953401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.955208 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.955188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:04.964914 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:04.964885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqztg\" (UniqueName: \"kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:05.418227 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:05.418191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:58:05.457673 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:05.457644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:05.460065 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:05.460038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:05.721399 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:05.721307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:05.997886 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:05.997861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 13:58:05.998841 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:58:05.998817 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a97a0d5_dba4_4957_9219_00645ba985fe.slice/crio-a8a8f24c2892ece203fedd8e9edb0df164719942bc1b106f344998f70d9373cb WatchSource:0}: Error finding container a8a8f24c2892ece203fedd8e9edb0df164719942bc1b106f344998f70d9373cb: Status 404 returned error can't find the container with id a8a8f24c2892ece203fedd8e9edb0df164719942bc1b106f344998f70d9373cb Sep 10 13:58:06.151253 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:06.151182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" event={"ID":"7a97a0d5-dba4-4957-9219-00645ba985fe","Type":"ContainerStarted","Data":"a8a8f24c2892ece203fedd8e9edb0df164719942bc1b106f344998f70d9373cb"} Sep 10 13:58:07.044586 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:07.044553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:07.107935 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:07.107909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:07.158720 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:07.158684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" event={"ID":"7a97a0d5-dba4-4957-9219-00645ba985fe","Type":"ContainerStarted","Data":"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54"} Sep 10 13:58:07.158872 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:07.158759 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:07.182922 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:07.182880 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" podStartSLOduration=2.773644632 podStartE2EDuration="3.182865444s" totalImagesPullingTime="409.220812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:07.179126345 +0000 UTC m=+1662.788911261" watchObservedRunningTime="2026-09-10 13:58:07.182865444 +0000 UTC m=+1662.792650358" Sep 10 13:58:11.806095 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:11.806049 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" podUID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" containerName="main" containerID="cri-o://639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e" gracePeriod=45 Sep 10 13:58:12.071217 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.071150 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:58:12.189040 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.189009 2559 generic.go:356] "Generic (PLEG): container finished" podID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" containerID="639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e" exitCode=0 Sep 10 13:58:12.189198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.189049 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" Sep 10 13:58:12.189198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.189091 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" event={"ID":"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6","Type":"ContainerDied","Data":"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e"} Sep 10 13:58:12.189198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.189138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7" event={"ID":"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6","Type":"ContainerDied","Data":"f31608d47b67a350ca0e4e6463c1c6842cadfa72518647d6c47a6b83236fcd5d"} Sep 10 13:58:12.189198 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.189161 2559 scope.go:117] "RemoveContainer" containerID="639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e" Sep 10 13:58:12.197966 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.197950 2559 scope.go:117] "RemoveContainer" containerID="639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e" Sep 10 13:58:12.198255 ip-10-0-133-208 kubenswrapper[2559]: E0910 13:58:12.198239 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e\": container with ID starting with 639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e not found: ID does not exist" containerID="639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e" Sep 10 13:58:12.198316 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.198267 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e"} err="failed to get container status \"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e\": rpc error: code = NotFound desc = could not find container \"639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e\": container with ID starting with 639f8f9c8d08058584b83e73d53683548fd293e4a8460cbf7b8a2c5c45bb6b9e not found: ID does not exist" Sep 10 13:58:12.213595 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213577 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213632 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213703 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213657 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213703 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213752 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nwj7\" (UniqueName: \"kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213810 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache\") pod \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\" (UID: \"f47a33ff-ed9c-40d5-9a15-20b4bf28fac6\") " Sep 10 13:58:12.213956 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213800 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:12.213956 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.213922 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home" (OuterVolumeSpecName: "home") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:12.214093 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.214067 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:12.214164 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.214147 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tmp-dir\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.214213 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.214170 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-home\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.215696 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.215670 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm" (OuterVolumeSpecName: "dshm") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:58:12.215793 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.215701 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 13:58:12.215839 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.215791 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7" (OuterVolumeSpecName: "kube-api-access-2nwj7") pod "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" (UID: "f47a33ff-ed9c-40d5-9a15-20b4bf28fac6"). InnerVolumeSpecName "kube-api-access-2nwj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:58:12.314764 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.314738 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-dshm\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.314764 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.314763 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-tls-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.314901 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.314783 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwj7\" (UniqueName: \"kubernetes.io/projected/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-kube-api-access-2nwj7\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.314901 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.314794 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6-model-cache\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 13:58:12.547438 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.547416 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:58:12.559076 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.559051 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-579ae1/rate-test-model-579ae1-kserve-8678577d99-cbcp7"] Sep 10 13:58:12.918697 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:12.918667 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" path="/var/lib/kubelet/pods/f47a33ff-ed9c-40d5-9a15-20b4bf28fac6/volumes" Sep 10 13:58:13.170776 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:13.170716 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 13:58:16.787355 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:16.787317 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:58:27.329231 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:27.329195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:27.768672 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:27.768634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:28.098940 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:28.098860 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:28.479716 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:28.479621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:28.593132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:28.593097 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:28.929026 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:28.928992 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:29.923145 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:29.923112 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:30.824194 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:30.824156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:31.269050 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:31.269016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:31.354831 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:31.354799 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:32.129995 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:32.129954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:32.931600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:32.931564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:34.103143 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:34.103101 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.022224 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.022190 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 13:58:35.022755 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.022735 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" containerName="main" Sep 10 13:58:35.022861 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.022758 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" containerName="main" Sep 10 13:58:35.022917 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.022895 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47a33ff-ed9c-40d5-9a15-20b4bf28fac6" containerName="main" Sep 10 13:58:35.026218 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.026196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.028630 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.028606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fc6ea5-serving-cert\"" Sep 10 13:58:35.035529 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.035506 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 13:58:35.050465 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.050430 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.151972 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.151942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.202055 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.202015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.202178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.202067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n6h\" (UniqueName: \"kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.202178 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.202090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.202262 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.202203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.251635 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.251601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.303132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.303071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.303132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.303106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n6h\" (UniqueName: \"kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.303132 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.303126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.303297 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.303194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.303852 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.303829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.305547 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.305525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.305646 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.305630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.313591 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.313570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n6h\" (UniqueName: \"kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h\") pod \"maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.339474 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.339452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:35.616594 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.616555 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 13:58:35.617488 ip-10-0-133-208 kubenswrapper[2559]: W0910 13:58:35.617459 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf4a260_21f5_4ada_aa3f_cd3b717701d5.slice/crio-0f57bd0fabceee7997a3f0054bfb7f42b5cfd4d12638377336c260dd55b9771f WatchSource:0}: Error finding container 0f57bd0fabceee7997a3f0054bfb7f42b5cfd4d12638377336c260dd55b9771f: Status 404 returned error can't find the container with id 0f57bd0fabceee7997a3f0054bfb7f42b5cfd4d12638377336c260dd55b9771f Sep 10 13:58:35.695957 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.695912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:35.797990 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:35.797956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:58:36.343745 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:36.343666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" event={"ID":"baf4a260-21f5-4ada-aa3f-cd3b717701d5","Type":"ContainerStarted","Data":"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971"} Sep 10 13:58:36.343745 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:36.343724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" event={"ID":"baf4a260-21f5-4ada-aa3f-cd3b717701d5","Type":"ContainerStarted","Data":"0f57bd0fabceee7997a3f0054bfb7f42b5cfd4d12638377336c260dd55b9771f"} Sep 10 13:58:36.344223 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:36.343759 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:36.364260 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:36.364219 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" podStartSLOduration=1.943738043 podStartE2EDuration="2.364203535s" totalImagesPullingTime="420.465492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:36.362289695 +0000 UTC m=+1691.972074608" watchObservedRunningTime="2026-09-10 13:58:36.364203535 +0000 UTC m=+1691.973988447" Sep 10 13:58:39.104921 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:39.104885 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:58:42.356083 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:42.356056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 13:58:59.513424 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:58:59.513298 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:00.762014 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:00.761984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:01.478955 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:01.478910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:01.967600 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:01.967556 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:03.181075 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:03.181029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:04.143992 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:04.143959 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:06.733741 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:06.733710 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 13:59:35.335431 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:35.335392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 13:59:39.685172 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:39.685128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 13:59:56.696129 ip-10-0-133-208 kubenswrapper[2559]: I0910 13:59:56.696098 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:00:03.268258 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:03.268203 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 14:00:03.986454 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:03.986420 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:00:03.990798 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:03.990777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:03.993515 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:03.993497 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ae1e0d-openshift-default-dockercfg-4v8pt\"" Sep 10 14:00:04.002177 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.002152 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:00:04.074161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.074476 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.074771 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.074895 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.074895 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.075017 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbth\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.075017 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.074983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.075132 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.075029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.075132 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.075069 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.085299 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.085264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:04.175923 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.175893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.175937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.175956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.175987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbth\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176437 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176437 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176342 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176558 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176634 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176689 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.176797 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.176781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.178481 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.178461 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.178601 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.178459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.179552 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.179520 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:04.191125 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.191107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbth\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.191426 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.191397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token\") pod \"e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.303028 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.303001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:04.601424 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.601266 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:00:04.604123 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:00:04.604093 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f024dd1_be65_4b50_97a4_e07005945c80.slice/crio-80d7535cc575b028d019bf68b6924a441d5c5c75eb73a66221fba9fb3bf422b3 WatchSource:0}: Error finding container 80d7535cc575b028d019bf68b6924a441d5c5c75eb73a66221fba9fb3bf422b3: Status 404 returned error can't find the container with id 80d7535cc575b028d019bf68b6924a441d5c5c75eb73a66221fba9fb3bf422b3 Sep 10 14:00:04.606282 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.606251 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:04.606343 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.606315 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:04.606415 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.606344 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:04.888617 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.888510 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerStarted","Data":"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e"} Sep 10 14:00:04.888617 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.888579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerStarted","Data":"80d7535cc575b028d019bf68b6924a441d5c5c75eb73a66221fba9fb3bf422b3"} Sep 10 14:00:04.917557 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:04.917500 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podStartSLOduration=1.917479546 podStartE2EDuration="1.917479546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:04.914259997 +0000 UTC m=+1780.524044931" watchObservedRunningTime="2026-09-10 14:00:04.917479546 +0000 UTC m=+1780.527264461" Sep 10 14:00:05.304212 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:05.304180 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:05.305747 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:05.305723 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:05.305867 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:05.305768 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:06.308770 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:06.308742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:06.368318 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:06.368285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:06.901349 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:06.901321 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:06.902331 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:06.902306 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:07.096703 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:07.096670 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:07.370041 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:07.370012 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:07.678925 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:07.678840 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:08.397979 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:08.397946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:09.422829 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:09.422785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:10.286130 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:10.286092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:11.084989 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.084956 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:11.089840 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.089815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.092256 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.092232 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ae1e0d-serving-cert\"" Sep 10 14:00:11.102445 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.102419 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:11.124304 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.124277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:11.194786 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.194760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:11.245605 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.245580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nrh\" (UniqueName: \"kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.245703 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.245650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.245703 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.245681 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.245817 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.245715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.296323 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.296279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:11.346602 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.346529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nrh\" (UniqueName: \"kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.346602 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.346580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.346762 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.346744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.346804 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.346789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.347391 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.347347 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.349117 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.349095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.349304 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.349283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.356935 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.356908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nrh\" (UniqueName: \"kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh\") pod \"maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.401449 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.401427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:11.681435 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.681407 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:11.682772 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:00:11.682746 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda640f5ff_a51e_4d97_980a_7e1c113dd178.slice/crio-653644e11dbc4198cc2800a888ee845d9853284c9a03d31bac3d54a2a55e0a87 WatchSource:0}: Error finding container 653644e11dbc4198cc2800a888ee845d9853284c9a03d31bac3d54a2a55e0a87: Status 404 returned error can't find the container with id 653644e11dbc4198cc2800a888ee845d9853284c9a03d31bac3d54a2a55e0a87 Sep 10 14:00:11.684538 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.684518 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:00:11.839353 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.839318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:11.932059 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:11.931983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" event={"ID":"a640f5ff-a51e-4d97-980a-7e1c113dd178","Type":"ContainerStarted","Data":"653644e11dbc4198cc2800a888ee845d9853284c9a03d31bac3d54a2a55e0a87"} Sep 10 14:00:13.289782 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:13.289744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:13.945155 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:13.944990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" event={"ID":"a640f5ff-a51e-4d97-980a-7e1c113dd178","Type":"ContainerStarted","Data":"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc"} Sep 10 14:00:13.945155 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:13.945125 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:13.965801 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:13.965755 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" podStartSLOduration=0.977755362 podStartE2EDuration="2.965740864s" totalImagesPullingTime="1.987985502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:13.964559282 +0000 UTC m=+1789.574344195" watchObservedRunningTime="2026-09-10 14:00:13.965740864 +0000 UTC m=+1789.575525774" Sep 10 14:00:17.629868 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:17.629787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:00:19.956488 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:19.956457 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:25.108465 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:25.108432 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 14:00:25.112965 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:25.112943 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 14:00:28.030962 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.030934 2559 generic.go:356] "Generic (PLEG): container finished" podID="6f024dd1-be65-4b50-97a4-e07005945c80" containerID="2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e" exitCode=0 Sep 10 14:00:28.031292 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.031000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerDied","Data":"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e"} Sep 10 14:00:28.031461 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.031444 2559 scope.go:117] "RemoveContainer" containerID="2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e" Sep 10 14:00:28.032598 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.032570 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:28.032679 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.032637 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:28.032679 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:28.032668 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 14:00:29.051238 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:29.051194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerStarted","Data":"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3"} Sep 10 14:00:29.303496 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:29.303409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:29.303704 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:29.303680 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:29.303792 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:29.303740 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:30.303298 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:30.303258 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:30.303781 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:30.303318 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:31.234078 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:31.234029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:31.303831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:31.303796 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:31.304189 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:31.303862 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:32.303941 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:32.303901 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:32.304452 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:32.303965 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:33.303651 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:33.303609 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 10 14:00:33.303904 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:33.303680 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 10 14:00:33.917344 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:33.917307 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:34.332024 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:34.331986 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:34.332618 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:34.332596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:34.333413 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:34.333357 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:00:34.835020 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:34.834991 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:35.731521 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:35.731488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:36.999278 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:36.999228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:37.095015 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:37.094984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:37.862638 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:37.862602 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:38.894866 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:38.894827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:39.821067 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:39.821034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:40.825663 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:40.825634 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:41.966522 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:41.966481 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:42.937743 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:42.937706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:43.908088 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:43.908051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:48.532013 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:48.531979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:49.341606 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.341575 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:49.342091 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.342060 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" podUID="a640f5ff-a51e-4d97-980a-7e1c113dd178" containerName="maas-api" containerID="cri-o://48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc" gracePeriod=30 Sep 10 14:00:49.377327 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.377284 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:49.475664 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.475632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:49.575863 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.575830 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:49.601802 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.601726 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:49.660658 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.660625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:49.694755 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.694728 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls\") pod \"a640f5ff-a51e-4d97-980a-7e1c113dd178\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " Sep 10 14:00:49.694963 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.694774 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42nrh\" (UniqueName: \"kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh\") pod \"a640f5ff-a51e-4d97-980a-7e1c113dd178\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " Sep 10 14:00:49.694963 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.694800 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs\") pod \"a640f5ff-a51e-4d97-980a-7e1c113dd178\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " Sep 10 14:00:49.694963 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.694852 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca\") pod \"a640f5ff-a51e-4d97-980a-7e1c113dd178\" (UID: \"a640f5ff-a51e-4d97-980a-7e1c113dd178\") " Sep 10 14:00:49.695429 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.695387 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a640f5ff-a51e-4d97-980a-7e1c113dd178" (UID: "a640f5ff-a51e-4d97-980a-7e1c113dd178"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:00:49.697140 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.697110 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh" (OuterVolumeSpecName: "kube-api-access-42nrh") pod "a640f5ff-a51e-4d97-980a-7e1c113dd178" (UID: "a640f5ff-a51e-4d97-980a-7e1c113dd178"). InnerVolumeSpecName "kube-api-access-42nrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:00:49.697430 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.697407 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a640f5ff-a51e-4d97-980a-7e1c113dd178" (UID: "a640f5ff-a51e-4d97-980a-7e1c113dd178"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:00:49.697509 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.697427 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a640f5ff-a51e-4d97-980a-7e1c113dd178" (UID: "a640f5ff-a51e-4d97-980a-7e1c113dd178"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:00:49.796535 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.796497 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:00:49.796535 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.796525 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nrh\" (UniqueName: \"kubernetes.io/projected/a640f5ff-a51e-4d97-980a-7e1c113dd178-kube-api-access-42nrh\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:00:49.796732 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.796539 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a640f5ff-a51e-4d97-980a-7e1c113dd178-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:00:49.796732 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:49.796565 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a640f5ff-a51e-4d97-980a-7e1c113dd178-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:00:50.108498 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.108456 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:50.177987 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.177947 2559 generic.go:356] "Generic (PLEG): container finished" podID="a640f5ff-a51e-4d97-980a-7e1c113dd178" containerID="48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc" exitCode=0 Sep 10 14:00:50.178161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.178012 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" Sep 10 14:00:50.178161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.178039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" event={"ID":"a640f5ff-a51e-4d97-980a-7e1c113dd178","Type":"ContainerDied","Data":"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc"} Sep 10 14:00:50.178161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.178124 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8" event={"ID":"a640f5ff-a51e-4d97-980a-7e1c113dd178","Type":"ContainerDied","Data":"653644e11dbc4198cc2800a888ee845d9853284c9a03d31bac3d54a2a55e0a87"} Sep 10 14:00:50.178161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.178150 2559 scope.go:117] "RemoveContainer" containerID="48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc" Sep 10 14:00:50.188261 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.188243 2559 scope.go:117] "RemoveContainer" containerID="48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc" Sep 10 14:00:50.188576 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:00:50.188550 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc\": container with ID starting with 48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc not found: ID does not exist" containerID="48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc" Sep 10 14:00:50.188679 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.188584 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc"} err="failed to get container status \"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc\": rpc error: code = NotFound desc = could not find container \"48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc\": container with ID starting with 48c34ac143ce2bc347d79cf7e5064bcc8eaa6e64522569b1435a085796059dfc not found: ID does not exist" Sep 10 14:00:50.209546 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.209487 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:50.209904 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.209882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:50.212581 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.212562 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ae1e0d-5b54bdc99-swsk8"] Sep 10 14:00:50.920178 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:50.920144 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a640f5ff-a51e-4d97-980a-7e1c113dd178" path="/var/lib/kubelet/pods/a640f5ff-a51e-4d97-980a-7e1c113dd178/volumes" Sep 10 14:00:51.911081 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:51.911044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:52.130696 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:52.130664 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:53.032064 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:53.032034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:54.192249 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:54.192215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:54.233420 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:54.233392 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:00:55.067169 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:55.067126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:55.153581 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:55.153548 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:55.254019 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:55.253981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:57.292104 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:57.292062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:57.354112 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:57.354017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:00:57.555617 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:57.555535 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:00:57.555763 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:57.555737 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" containerID="cri-o://86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3" gracePeriod=30 Sep 10 14:00:57.583193 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:00:57.583162 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:01.566060 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:01.566010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:02.807480 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.807456 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:01:02.917576 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917506 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917576 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917546 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917576 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917565 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917601 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917683 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917723 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917762 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.917831 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917808 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhbth\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.918097 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket\") pod \"6f024dd1-be65-4b50-97a4-e07005945c80\" (UID: \"6f024dd1-be65-4b50-97a4-e07005945c80\") " Sep 10 14:01:02.918097 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917840 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:02.918097 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917854 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:02.918097 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.917956 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:02.918291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918154 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:02.918291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918203 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:02.918291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918243 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:02.918291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918265 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:02.918291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918280 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:02.918538 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.918293 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:02.919953 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.919923 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:02.920239 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.920214 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:01:02.920337 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.920307 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth" (OuterVolumeSpecName: "kube-api-access-fhbth") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "kube-api-access-fhbth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:02.920337 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:02.920307 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f024dd1-be65-4b50-97a4-e07005945c80" (UID: "6f024dd1-be65-4b50-97a4-e07005945c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:01:03.019421 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.019396 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f024dd1-be65-4b50-97a4-e07005945c80-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:03.019421 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.019417 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:03.019567 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.019426 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f024dd1-be65-4b50-97a4-e07005945c80-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:03.019567 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.019435 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhbth\" (UniqueName: \"kubernetes.io/projected/6f024dd1-be65-4b50-97a4-e07005945c80-kube-api-access-fhbth\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:03.019567 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.019445 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f024dd1-be65-4b50-97a4-e07005945c80-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:03.259914 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.259884 2559 generic.go:356] "Generic (PLEG): container finished" podID="6f024dd1-be65-4b50-97a4-e07005945c80" containerID="86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3" exitCode=0 Sep 10 14:01:03.260039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.259938 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" Sep 10 14:01:03.260039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.259963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerDied","Data":"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3"} Sep 10 14:01:03.260039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.260016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj" event={"ID":"6f024dd1-be65-4b50-97a4-e07005945c80","Type":"ContainerDied","Data":"80d7535cc575b028d019bf68b6924a441d5c5c75eb73a66221fba9fb3bf422b3"} Sep 10 14:01:03.260039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.260039 2559 scope.go:117] "RemoveContainer" containerID="86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3" Sep 10 14:01:03.270314 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.270296 2559 scope.go:117] "RemoveContainer" containerID="2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e" Sep 10 14:01:03.278746 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.278730 2559 scope.go:117] "RemoveContainer" containerID="86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3" Sep 10 14:01:03.279016 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:01:03.278991 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3\": container with ID starting with 86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3 not found: ID does not exist" containerID="86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3" Sep 10 14:01:03.279065 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.279014 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3"} err="failed to get container status \"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3\": rpc error: code = NotFound desc = could not find container \"86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3\": container with ID starting with 86c73760fdeb9405219cf00d705bfa166f2435bb5b1d4b8c80f59182bc38d4b3 not found: ID does not exist" Sep 10 14:01:03.279065 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.279030 2559 scope.go:117] "RemoveContainer" containerID="2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e" Sep 10 14:01:03.279265 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:01:03.279247 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e\": container with ID starting with 2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e not found: ID does not exist" containerID="2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e" Sep 10 14:01:03.279301 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.279273 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e"} err="failed to get container status \"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e\": rpc error: code = NotFound desc = could not find container \"2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e\": container with ID starting with 2378fa13ad7bbef2a71faef7c9322e8a6143160b8acad92141c3b674c96c759e not found: ID does not exist" Sep 10 14:01:03.293461 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.293438 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:01:03.303220 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:03.303190 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ae1e0d-openshift-default-f647fb6f6-7t5pj"] Sep 10 14:01:04.068324 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:04.068274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:04.171285 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:04.171249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:04.515692 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:04.515661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:04.919500 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:04.919421 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" path="/var/lib/kubelet/pods/6f024dd1-be65-4b50-97a4-e07005945c80/volumes" Sep 10 14:01:05.596308 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:05.596274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:06.475681 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:06.475650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:15.097393 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:15.097346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:01:17.236422 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.236342 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 14:01:17.236895 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.236656 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" podUID="7a97a0d5-dba4-4957-9219-00645ba985fe" containerName="maas-api" containerID="cri-o://749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54" gracePeriod=30 Sep 10 14:01:17.294187 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.294155 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:17.430957 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.430851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:17.497751 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.497689 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 14:01:17.529149 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.529116 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:17.634988 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.634129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:17.638337 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.638312 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs\") pod \"7a97a0d5-dba4-4957-9219-00645ba985fe\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " Sep 10 14:01:17.638503 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.638411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqztg\" (UniqueName: \"kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg\") pod \"7a97a0d5-dba4-4957-9219-00645ba985fe\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " Sep 10 14:01:17.638572 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.638505 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") pod \"7a97a0d5-dba4-4957-9219-00645ba985fe\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " Sep 10 14:01:17.638572 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.638548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca\") pod \"7a97a0d5-dba4-4957-9219-00645ba985fe\" (UID: \"7a97a0d5-dba4-4957-9219-00645ba985fe\") " Sep 10 14:01:17.638986 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.638954 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a97a0d5-dba4-4957-9219-00645ba985fe" (UID: "7a97a0d5-dba4-4957-9219-00645ba985fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:01:17.641172 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.641146 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a97a0d5-dba4-4957-9219-00645ba985fe" (UID: "7a97a0d5-dba4-4957-9219-00645ba985fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:17.641265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.641162 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a97a0d5-dba4-4957-9219-00645ba985fe" (UID: "7a97a0d5-dba4-4957-9219-00645ba985fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:01:17.641265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.641165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg" (OuterVolumeSpecName: "kube-api-access-zqztg") pod "7a97a0d5-dba4-4957-9219-00645ba985fe" (UID: "7a97a0d5-dba4-4957-9219-00645ba985fe"). InnerVolumeSpecName "kube-api-access-zqztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:01:17.739915 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.739826 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqztg\" (UniqueName: \"kubernetes.io/projected/7a97a0d5-dba4-4957-9219-00645ba985fe-kube-api-access-zqztg\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:17.739915 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.739863 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:17.739915 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.739878 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a97a0d5-dba4-4957-9219-00645ba985fe-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:17.739915 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.739891 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a97a0d5-dba4-4957-9219-00645ba985fe-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:01:17.745196 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:17.745164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:18.124536 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.124503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:18.226352 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.226322 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:18.350879 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.350843 2559 generic.go:356] "Generic (PLEG): container finished" podID="7a97a0d5-dba4-4957-9219-00645ba985fe" containerID="749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54" exitCode=0 Sep 10 14:01:18.351257 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.350885 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" Sep 10 14:01:18.351257 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.350904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" event={"ID":"7a97a0d5-dba4-4957-9219-00645ba985fe","Type":"ContainerDied","Data":"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54"} Sep 10 14:01:18.351257 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.350952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7" event={"ID":"7a97a0d5-dba4-4957-9219-00645ba985fe","Type":"ContainerDied","Data":"a8a8f24c2892ece203fedd8e9edb0df164719942bc1b106f344998f70d9373cb"} Sep 10 14:01:18.351257 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.350977 2559 scope.go:117] "RemoveContainer" containerID="749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54" Sep 10 14:01:18.361836 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.361821 2559 scope.go:117] "RemoveContainer" containerID="749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54" Sep 10 14:01:18.362087 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:01:18.362069 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54\": container with ID starting with 749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54 not found: ID does not exist" containerID="749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54" Sep 10 14:01:18.362134 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.362094 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54"} err="failed to get container status \"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54\": rpc error: code = NotFound desc = could not find container \"749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54\": container with ID starting with 749078119cf9da9b20d1377b169fbf24c90e01ff816dbf04141db6e4aea71f54 not found: ID does not exist" Sep 10 14:01:18.386872 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.386808 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 14:01:18.390205 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.390179 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fe383-74987c5db5-lqrr7"] Sep 10 14:01:18.918228 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:18.918197 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a97a0d5-dba4-4957-9219-00645ba985fe" path="/var/lib/kubelet/pods/7a97a0d5-dba4-4957-9219-00645ba985fe/volumes" Sep 10 14:01:19.312778 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:19.312741 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:20.638640 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:20.638603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:21.799892 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:21.799854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:25.312666 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:25.312626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:25.395266 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:25.395239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:28.495511 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:28.495477 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:01:39.655655 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:39.655617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:42.215664 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:42.215624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:01:50.549845 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:01:50.549771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:09.694628 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:09.694589 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:11.773688 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:11.773653 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:16.851191 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:16.851143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:16.956232 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:16.956179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:17.811337 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:17.811278 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:18.665554 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:18.665517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:19.921005 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:19.920972 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:20.662188 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:20.662154 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:27.051292 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.051253 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 14:02:27.051856 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.051520 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" podUID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" containerName="maas-api" containerID="cri-o://f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971" gracePeriod=30 Sep 10 14:02:27.120055 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.120017 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:27.282403 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.282354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:27.316343 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.316292 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 14:02:27.448840 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.448812 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca\") pod \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " Sep 10 14:02:27.448982 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.448883 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls\") pod \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " Sep 10 14:02:27.448982 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.448921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9n6h\" (UniqueName: \"kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h\") pod \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " Sep 10 14:02:27.448982 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.448948 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs\") pod \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\" (UID: \"baf4a260-21f5-4ada-aa3f-cd3b717701d5\") " Sep 10 14:02:27.449244 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.449215 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "baf4a260-21f5-4ada-aa3f-cd3b717701d5" (UID: "baf4a260-21f5-4ada-aa3f-cd3b717701d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:02:27.451039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.451011 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h" (OuterVolumeSpecName: "kube-api-access-c9n6h") pod "baf4a260-21f5-4ada-aa3f-cd3b717701d5" (UID: "baf4a260-21f5-4ada-aa3f-cd3b717701d5"). InnerVolumeSpecName "kube-api-access-c9n6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:02:27.451039 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.451019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "baf4a260-21f5-4ada-aa3f-cd3b717701d5" (UID: "baf4a260-21f5-4ada-aa3f-cd3b717701d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:02:27.451182 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.451067 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "baf4a260-21f5-4ada-aa3f-cd3b717701d5" (UID: "baf4a260-21f5-4ada-aa3f-cd3b717701d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:02:27.550435 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.550404 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:27.550435 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.550433 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9n6h\" (UniqueName: \"kubernetes.io/projected/baf4a260-21f5-4ada-aa3f-cd3b717701d5-kube-api-access-c9n6h\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:27.550435 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.550442 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf4a260-21f5-4ada-aa3f-cd3b717701d5-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:27.550654 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.550450 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf4a260-21f5-4ada-aa3f-cd3b717701d5-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:27.771693 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.771659 2559 generic.go:356] "Generic (PLEG): container finished" podID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" containerID="f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971" exitCode=0 Sep 10 14:02:27.771871 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.771702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" Sep 10 14:02:27.771871 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.771740 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" event={"ID":"baf4a260-21f5-4ada-aa3f-cd3b717701d5","Type":"ContainerDied","Data":"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971"} Sep 10 14:02:27.771871 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.771802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh" event={"ID":"baf4a260-21f5-4ada-aa3f-cd3b717701d5","Type":"ContainerDied","Data":"0f57bd0fabceee7997a3f0054bfb7f42b5cfd4d12638377336c260dd55b9771f"} Sep 10 14:02:27.771871 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.771831 2559 scope.go:117] "RemoveContainer" containerID="f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971" Sep 10 14:02:27.782274 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.782257 2559 scope.go:117] "RemoveContainer" containerID="f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971" Sep 10 14:02:27.782590 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:02:27.782569 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971\": container with ID starting with f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971 not found: ID does not exist" containerID="f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971" Sep 10 14:02:27.782694 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.782596 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971"} err="failed to get container status \"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971\": rpc error: code = NotFound desc = could not find container \"f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971\": container with ID starting with f86853f6131063bf4eb9f920b2caacefa0247d52f1c725e0f95ab2896ee53971 not found: ID does not exist" Sep 10 14:02:27.799007 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.798954 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 14:02:27.801737 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:27.801713 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc6ea5-d8f5f7cbf-4x6wh"] Sep 10 14:02:28.872343 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:28.872308 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:28.922555 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:28.920264 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" path="/var/lib/kubelet/pods/baf4a260-21f5-4ada-aa3f-cd3b717701d5/volumes" Sep 10 14:02:28.970759 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:28.970724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:29.188138 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:29.188065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:30.180796 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:30.180758 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:30.514266 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:30.514238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:02:35.228886 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:35.228846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:35.310917 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:35.310887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:35.538844 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:35.538814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:37.095110 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:37.095073 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:38.039638 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:38.039598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:39.019724 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:39.019685 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:46.539468 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.539429 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 14:02:46.539846 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.539704 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" podUID="e0855d13-a1c0-4e38-9c2b-2134f534796b" containerName="maas-api" containerID="cri-o://f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984" gracePeriod=30 Sep 10 14:02:46.785595 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.785571 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 14:02:46.890242 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.890168 2559 generic.go:356] "Generic (PLEG): container finished" podID="e0855d13-a1c0-4e38-9c2b-2134f534796b" containerID="f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984" exitCode=0 Sep 10 14:02:46.890242 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.890217 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" Sep 10 14:02:46.890528 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.890250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" event={"ID":"e0855d13-a1c0-4e38-9c2b-2134f534796b","Type":"ContainerDied","Data":"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984"} Sep 10 14:02:46.890528 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.890307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn" event={"ID":"e0855d13-a1c0-4e38-9c2b-2134f534796b","Type":"ContainerDied","Data":"68220f327bc1b5149290cd20b751951fec2407326097e17f132f0e384ed08188"} Sep 10 14:02:46.890528 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.890330 2559 scope.go:117] "RemoveContainer" containerID="f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984" Sep 10 14:02:46.901702 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.901676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqmv9\" (UniqueName: \"kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9\") pod \"e0855d13-a1c0-4e38-9c2b-2134f534796b\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " Sep 10 14:02:46.901983 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.901958 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") pod \"e0855d13-a1c0-4e38-9c2b-2134f534796b\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " Sep 10 14:02:46.902133 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.901999 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca\") pod \"e0855d13-a1c0-4e38-9c2b-2134f534796b\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " Sep 10 14:02:46.902133 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.902055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs\") pod \"e0855d13-a1c0-4e38-9c2b-2134f534796b\" (UID: \"e0855d13-a1c0-4e38-9c2b-2134f534796b\") " Sep 10 14:02:46.902706 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.902411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0855d13-a1c0-4e38-9c2b-2134f534796b" (UID: "e0855d13-a1c0-4e38-9c2b-2134f534796b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:02:46.904161 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.904131 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9" (OuterVolumeSpecName: "kube-api-access-zqmv9") pod "e0855d13-a1c0-4e38-9c2b-2134f534796b" (UID: "e0855d13-a1c0-4e38-9c2b-2134f534796b"). InnerVolumeSpecName "kube-api-access-zqmv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:02:46.904520 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.904497 2559 scope.go:117] "RemoveContainer" containerID="f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984" Sep 10 14:02:46.904636 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.904535 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0855d13-a1c0-4e38-9c2b-2134f534796b" (UID: "e0855d13-a1c0-4e38-9c2b-2134f534796b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:02:46.904832 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:02:46.904803 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984\": container with ID starting with f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984 not found: ID does not exist" containerID="f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984" Sep 10 14:02:46.904921 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.904834 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984"} err="failed to get container status \"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984\": rpc error: code = NotFound desc = could not find container \"f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984\": container with ID starting with f3f765f0b56a48fe53da96b3143e7f98218689fb7ee1af5ac052955731890984 not found: ID does not exist" Sep 10 14:02:46.904921 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:46.904846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e0855d13-a1c0-4e38-9c2b-2134f534796b" (UID: "e0855d13-a1c0-4e38-9c2b-2134f534796b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:02:47.002876 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.002839 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmv9\" (UniqueName: \"kubernetes.io/projected/e0855d13-a1c0-4e38-9c2b-2134f534796b-kube-api-access-zqmv9\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:47.002876 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.002870 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:47.002876 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.002881 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0855d13-a1c0-4e38-9c2b-2134f534796b-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:47.003111 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.002889 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0855d13-a1c0-4e38-9c2b-2134f534796b-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:02:47.218268 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.218189 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 14:02:47.226086 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.226060 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-579ae1-86686d5c-t86tn"] Sep 10 14:02:47.911065 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:47.911039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:48.055494 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:48.055460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:48.454654 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:48.454617 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:48.917859 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:48.917825 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0855d13-a1c0-4e38-9c2b-2134f534796b" path="/var/lib/kubelet/pods/e0855d13-a1c0-4e38-9c2b-2134f534796b/volumes" Sep 10 14:02:49.819428 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:49.819393 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:02:54.931484 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:54.931451 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 14:02:54.931977 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:54.931652 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" containerID="cri-o://cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58" gracePeriod=30 Sep 10 14:02:56.083657 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:56.083609 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:56.225654 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:56.225621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:56.790424 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:56.790388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:02:58.012164 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:02:58.012117 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:00.188318 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.188290 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 14:03:00.219238 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219175 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219238 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219219 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219446 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219255 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219446 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6gj5\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219446 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219343 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219446 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219657 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219456 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219657 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219492 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219657 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy\") pod \"4de37bf8-5152-418d-abf8-64516c1bc578\" (UID: \"4de37bf8-5152-418d-abf8-64516c1bc578\") " Sep 10 14:03:00.219657 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219580 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:00.219847 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219693 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:00.219847 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219813 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data" (OuterVolumeSpecName: "istio-data") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:00.219847 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219828 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:00.220009 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219953 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.220009 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219973 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de37bf8-5152-418d-abf8-64516c1bc578-istiod-ca-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.220009 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219987 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-workload-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.220009 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.220000 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-data\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.220009 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.219968 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:00.221428 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.221396 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:03:00.221533 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.221485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5" (OuterVolumeSpecName: "kube-api-access-s6gj5") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "kube-api-access-s6gj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:00.221863 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.221839 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token" (OuterVolumeSpecName: "istio-token") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:00.221937 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.221916 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4de37bf8-5152-418d-abf8-64516c1bc578" (UID: "4de37bf8-5152-418d-abf8-64516c1bc578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:03:00.321338 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.321313 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-istio-token\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.321338 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.321338 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de37bf8-5152-418d-abf8-64516c1bc578-istio-podinfo\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.321504 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.321353 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gj5\" (UniqueName: \"kubernetes.io/projected/4de37bf8-5152-418d-abf8-64516c1bc578-kube-api-access-s6gj5\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.321504 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.321386 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-credential-socket\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.321504 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.321395 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de37bf8-5152-418d-abf8-64516c1bc578-istio-envoy\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:00.971596 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.971567 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de37bf8-5152-418d-abf8-64516c1bc578" containerID="cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58" exitCode=0 Sep 10 14:03:00.971729 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.971613 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" Sep 10 14:03:00.971729 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.971663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerDied","Data":"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58"} Sep 10 14:03:00.971729 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.971723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j" event={"ID":"4de37bf8-5152-418d-abf8-64516c1bc578","Type":"ContainerDied","Data":"44b89059f8a96e06811084f7761de3bae887129af2550cdbca9fb3a1d3522e14"} Sep 10 14:03:00.971896 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.971754 2559 scope.go:117] "RemoveContainer" containerID="cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58" Sep 10 14:03:00.981231 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.981212 2559 scope.go:117] "RemoveContainer" containerID="2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80" Sep 10 14:03:00.989983 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.989967 2559 scope.go:117] "RemoveContainer" containerID="cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58" Sep 10 14:03:00.990253 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:03:00.990228 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58\": container with ID starting with cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58 not found: ID does not exist" containerID="cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58" Sep 10 14:03:00.990361 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.990257 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58"} err="failed to get container status \"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58\": rpc error: code = NotFound desc = could not find container \"cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58\": container with ID starting with cc06bd19de147f58bb701a50288fa95d8749c6a41d07cdf593b8875b5aeb1e58 not found: ID does not exist" Sep 10 14:03:00.990361 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.990279 2559 scope.go:117] "RemoveContainer" containerID="2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80" Sep 10 14:03:00.990607 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:03:00.990590 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80\": container with ID starting with 2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80 not found: ID does not exist" containerID="2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80" Sep 10 14:03:00.990667 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:00.990626 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80"} err="failed to get container status \"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80\": rpc error: code = NotFound desc = could not find container \"2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80\": container with ID starting with 2712534d7263ed7cd65401a2b2aaefaa29026200a4ebf0463cc2f8469f11bb80 not found: ID does not exist" Sep 10 14:03:01.004405 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:01.004382 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 14:03:01.017211 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:01.017191 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-579ae1-openshift-default-5c98577844-rqk8j"] Sep 10 14:03:02.918186 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:02.918154 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" path="/var/lib/kubelet/pods/4de37bf8-5152-418d-abf8-64516c1bc578/volumes" Sep 10 14:03:06.171490 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.171455 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 14:03:06.171968 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.171691 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" podUID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" containerName="maas-api" containerID="cri-o://5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443" gracePeriod=30 Sep 10 14:03:06.220603 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.220573 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:06.401767 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.401740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:06.441982 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.441955 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 14:03:06.467828 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.467805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6xkf\" (UniqueName: \"kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf\") pod \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " Sep 10 14:03:06.467925 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.467861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs\") pod \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " Sep 10 14:03:06.467996 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.467976 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca\") pod \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " Sep 10 14:03:06.468122 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.468105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") pod \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\" (UID: \"a4cdaa2d-9abb-4bd5-a649-b6f709680436\") " Sep 10 14:03:06.468354 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.468325 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4cdaa2d-9abb-4bd5-a649-b6f709680436" (UID: "a4cdaa2d-9abb-4bd5-a649-b6f709680436"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:03:06.469832 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.469807 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf" (OuterVolumeSpecName: "kube-api-access-r6xkf") pod "a4cdaa2d-9abb-4bd5-a649-b6f709680436" (UID: "a4cdaa2d-9abb-4bd5-a649-b6f709680436"). InnerVolumeSpecName "kube-api-access-r6xkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:03:06.469969 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.469946 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4cdaa2d-9abb-4bd5-a649-b6f709680436" (UID: "a4cdaa2d-9abb-4bd5-a649-b6f709680436"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:06.470013 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.469962 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4cdaa2d-9abb-4bd5-a649-b6f709680436" (UID: "a4cdaa2d-9abb-4bd5-a649-b6f709680436"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:03:06.505409 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.505343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:06.569678 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.569655 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:06.569678 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.569676 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4cdaa2d-9abb-4bd5-a649-b6f709680436-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:06.569816 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.569688 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4cdaa2d-9abb-4bd5-a649-b6f709680436-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:06.569816 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.569698 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6xkf\" (UniqueName: \"kubernetes.io/projected/a4cdaa2d-9abb-4bd5-a649-b6f709680436-kube-api-access-r6xkf\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:03:06.599387 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.599341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:06.668006 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:06.667969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:07.014796 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.014766 2559 generic.go:356] "Generic (PLEG): container finished" podID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" containerID="5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443" exitCode=0 Sep 10 14:03:07.014940 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.014810 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" Sep 10 14:03:07.014940 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.014851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" event={"ID":"a4cdaa2d-9abb-4bd5-a649-b6f709680436","Type":"ContainerDied","Data":"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443"} Sep 10 14:03:07.014940 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.014901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4" event={"ID":"a4cdaa2d-9abb-4bd5-a649-b6f709680436","Type":"ContainerDied","Data":"190275ae517aded3f00e6c2fa40e3e8155fbaecb07e77b28e65de10d127b7d11"} Sep 10 14:03:07.014940 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.014927 2559 scope.go:117] "RemoveContainer" containerID="5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443" Sep 10 14:03:07.024296 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.024280 2559 scope.go:117] "RemoveContainer" containerID="5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443" Sep 10 14:03:07.024564 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:03:07.024541 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443\": container with ID starting with 5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443 not found: ID does not exist" containerID="5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443" Sep 10 14:03:07.024629 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.024572 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443"} err="failed to get container status \"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443\": rpc error: code = NotFound desc = could not find container \"5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443\": container with ID starting with 5a6d5e45b4ee53b589adcaddec519db2044282a6b79849984c727b2fa9450443 not found: ID does not exist" Sep 10 14:03:07.056959 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.056928 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 14:03:07.065678 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.065658 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d5c012-854dd5c779-bvss4"] Sep 10 14:03:07.084491 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.084469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:07.239921 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:07.239846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:08.919765 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:08.919724 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" path="/var/lib/kubelet/pods/a4cdaa2d-9abb-4bd5-a649-b6f709680436/volumes" Sep 10 14:03:08.964953 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:08.964921 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:03:16.411034 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:16.410990 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:16.638482 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:16.638446 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:16.809914 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:16.809873 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:03:20.938610 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:03:20.938577 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:04:42.789668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:04:42.789637 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:04:43.653179 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:04:43.653144 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:04:45.165141 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:04:45.165107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:09.395705 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:09.395669 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:05:16.470301 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:16.470264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:05:17.491224 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:17.491184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:05:25.148828 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:25.148798 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 14:05:25.153274 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:25.153248 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 14:05:25.824093 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:25.824061 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:05:43.639669 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:05:43.639626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:07:02.160518 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:02.160476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:11.547255 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:11.547218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:12.155740 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:12.155709 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:07:20.505857 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:20.505818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:07:28.924850 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:28.924803 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:07:35.360190 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360159 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f7c4c7dd-vm7h4"] Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360533 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360545 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360554 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360560 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360567 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0855d13-a1c0-4e38-9c2b-2134f534796b" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360574 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0855d13-a1c0-4e38-9c2b-2134f534796b" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360583 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360588 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360599 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360604 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360613 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a97a0d5-dba4-4957-9219-00645ba985fe" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360618 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a97a0d5-dba4-4957-9219-00645ba985fe" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360624 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a640f5ff-a51e-4d97-980a-7e1c113dd178" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360629 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a640f5ff-a51e-4d97-980a-7e1c113dd178" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360635 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360640 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360648 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" containerName="maas-api" Sep 10 14:07:35.360668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360653 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360706 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf4a260-21f5-4ada-aa3f-cd3b717701d5" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360714 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360721 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f024dd1-be65-4b50-97a4-e07005945c80" containerName="istio-proxy" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360726 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360730 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0855d13-a1c0-4e38-9c2b-2134f534796b" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360736 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a97a0d5-dba4-4957-9219-00645ba985fe" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360742 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4cdaa2d-9abb-4bd5-a649-b6f709680436" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360749 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a640f5ff-a51e-4d97-980a-7e1c113dd178" containerName="maas-api" Sep 10 14:07:35.361265 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.360753 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de37bf8-5152-418d-abf8-64516c1bc578" containerName="istio-proxy" Sep 10 14:07:35.363757 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.363734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.378156 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.378132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f7c4c7dd-vm7h4"] Sep 10 14:07:35.465350 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.465324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:35.475871 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.475846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-oauth-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.475939 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.475884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfb5\" (UniqueName: \"kubernetes.io/projected/0d4637f1-7a0b-427b-b354-03aaa39ca196-kube-api-access-bzfb5\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.475939 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.475906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-trusted-ca-bundle\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.476030 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.475966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.476030 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.475987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-oauth-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.476110 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.476032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-service-ca\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.476110 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.476082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.576887 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.576861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-trusted-ca-bundle\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.576981 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.576900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.576981 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.576921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-oauth-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.576981 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.576971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-service-ca\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577152 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577152 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-oauth-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577438 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfb5\" (UniqueName: \"kubernetes.io/projected/0d4637f1-7a0b-427b-b354-03aaa39ca196-kube-api-access-bzfb5\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577762 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-service-ca\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577870 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577870 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-oauth-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.577966 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.577908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d4637f1-7a0b-427b-b354-03aaa39ca196-trusted-ca-bundle\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.578835 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.578808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:35.579624 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.579606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-serving-cert\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.579718 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.579676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d4637f1-7a0b-427b-b354-03aaa39ca196-console-oauth-config\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.586846 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.586827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfb5\" (UniqueName: \"kubernetes.io/projected/0d4637f1-7a0b-427b-b354-03aaa39ca196-kube-api-access-bzfb5\") pod \"console-84f7c4c7dd-vm7h4\" (UID: \"0d4637f1-7a0b-427b-b354-03aaa39ca196\") " pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.674843 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.674783 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:35.955789 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:35.955761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f7c4c7dd-vm7h4"] Sep 10 14:07:35.957000 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:07:35.956968 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4637f1_7a0b_427b_b354_03aaa39ca196.slice/crio-86d8c869ff341f4809296c01e831bb664e93affad6e877cd55d1cf38cfcd41ba WatchSource:0}: Error finding container 86d8c869ff341f4809296c01e831bb664e93affad6e877cd55d1cf38cfcd41ba: Status 404 returned error can't find the container with id 86d8c869ff341f4809296c01e831bb664e93affad6e877cd55d1cf38cfcd41ba Sep 10 14:07:36.551291 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:36.551250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f7c4c7dd-vm7h4" event={"ID":"0d4637f1-7a0b-427b-b354-03aaa39ca196","Type":"ContainerStarted","Data":"c2cc5855e54a5106948d299815fad716a27ee838c48662d81e702b388bb2f63d"} Sep 10 14:07:36.551791 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:36.551305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f7c4c7dd-vm7h4" event={"ID":"0d4637f1-7a0b-427b-b354-03aaa39ca196","Type":"ContainerStarted","Data":"86d8c869ff341f4809296c01e831bb664e93affad6e877cd55d1cf38cfcd41ba"} Sep 10 14:07:36.576699 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:36.576647 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f7c4c7dd-vm7h4" podStartSLOduration=1.5766313410000001 podStartE2EDuration="1.576631341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:07:36.573496475 +0000 UTC m=+2232.183281388" watchObservedRunningTime="2026-09-10 14:07:36.576631341 +0000 UTC m=+2232.186416253" Sep 10 14:07:38.244015 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:38.243979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:41.112258 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:41.112220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:07:45.675765 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:45.675729 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:45.676144 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:45.675991 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:45.680894 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:45.680872 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:46.612793 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:46.612761 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f7c4c7dd-vm7h4" Sep 10 14:07:46.676530 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:46.676493 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 14:07:47.631105 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:47.631028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:47.715344 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:47.715310 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:07:51.158961 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:07:51.158927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:08:07.177268 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:07.177231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:08:10.271506 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:10.271469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:08:11.699404 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:11.699345 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7658ccd565-vp6f9" podUID="56ec93e6-f123-491a-948e-2ab0db48e639" containerName="console" containerID="cri-o://2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d" gracePeriod=15 Sep 10 14:08:11.973836 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:11.973805 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063234 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063293 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063332 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrnc4\" (UniqueName: \"kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.063627 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.063553 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca\") pod \"56ec93e6-f123-491a-948e-2ab0db48e639\" (UID: \"56ec93e6-f123-491a-948e-2ab0db48e639\") " Sep 10 14:08:12.064464 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.064225 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca" (OuterVolumeSpecName: "service-ca") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:08:12.065395 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.064639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config" (OuterVolumeSpecName: "console-config") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:08:12.065954 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.065865 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:08:12.065954 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.065884 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:08:12.068045 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.067998 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4" (OuterVolumeSpecName: "kube-api-access-qrnc4") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "kube-api-access-qrnc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:08:12.070075 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.070053 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:08:12.070326 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.070302 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "56ec93e6-f123-491a-948e-2ab0db48e639" (UID: "56ec93e6-f123-491a-948e-2ab0db48e639"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:08:12.165129 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165099 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165129 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165123 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-trusted-ca-bundle\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165129 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165133 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165322 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165142 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56ec93e6-f123-491a-948e-2ab0db48e639-console-oauth-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165322 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165151 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-console-config\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165322 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165159 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnc4\" (UniqueName: \"kubernetes.io/projected/56ec93e6-f123-491a-948e-2ab0db48e639-kube-api-access-qrnc4\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.165322 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.165168 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56ec93e6-f123-491a-948e-2ab0db48e639-oauth-serving-cert\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:08:12.766560 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.766522 2559 generic.go:356] "Generic (PLEG): container finished" podID="56ec93e6-f123-491a-948e-2ab0db48e639" containerID="2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d" exitCode=0 Sep 10 14:08:12.766960 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.766582 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7658ccd565-vp6f9" Sep 10 14:08:12.766960 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.766610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7658ccd565-vp6f9" event={"ID":"56ec93e6-f123-491a-948e-2ab0db48e639","Type":"ContainerDied","Data":"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d"} Sep 10 14:08:12.766960 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.766690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7658ccd565-vp6f9" event={"ID":"56ec93e6-f123-491a-948e-2ab0db48e639","Type":"ContainerDied","Data":"7c02bcd7a4e09a59493eb04d163056cad377af707a15a8e344fa506dab309c21"} Sep 10 14:08:12.766960 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.766725 2559 scope.go:117] "RemoveContainer" containerID="2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d" Sep 10 14:08:12.776839 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.776820 2559 scope.go:117] "RemoveContainer" containerID="2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d" Sep 10 14:08:12.777134 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:08:12.777115 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d\": container with ID starting with 2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d not found: ID does not exist" containerID="2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d" Sep 10 14:08:12.777208 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.777149 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d"} err="failed to get container status \"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d\": rpc error: code = NotFound desc = could not find container \"2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d\": container with ID starting with 2be6fe412a42d37db8133ab4de259bf3ad7c0c6d128c1ea900695e094915210d not found: ID does not exist" Sep 10 14:08:12.876470 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.876449 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 14:08:12.882233 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.882209 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7658ccd565-vp6f9"] Sep 10 14:08:12.918525 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:12.918504 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ec93e6-f123-491a-948e-2ab0db48e639" path="/var/lib/kubelet/pods/56ec93e6-f123-491a-948e-2ab0db48e639/volumes" Sep 10 14:08:29.068121 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:29.068054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:08:31.108446 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:31.108413 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:08:31.166075 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:31.166034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:08:39.119442 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:39.119407 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:08:45.123587 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:45.123544 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:08:52.397775 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.397740 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:08:52.684077 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.684003 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:08:52.684382 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.684355 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ec93e6-f123-491a-948e-2ab0db48e639" containerName="console" Sep 10 14:08:52.684438 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.684383 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ec93e6-f123-491a-948e-2ab0db48e639" containerName="console" Sep 10 14:08:52.684476 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.684465 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ec93e6-f123-491a-948e-2ab0db48e639" containerName="console" Sep 10 14:08:52.687722 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.687704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.691544 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.691520 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 14:08:52.691695 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.691567 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 14:08:52.691695 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.691612 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-97x99\"" Sep 10 14:08:52.748785 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.748758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:08:52.788975 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.788952 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddxc\" (UniqueName: \"kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.789087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.788995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.789087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.789029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.789087 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.789069 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.823853 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.823826 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6"] Sep 10 14:08:52.827167 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.827149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:52.855666 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.855646 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6"] Sep 10 14:08:52.890070 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.890049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.890184 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.890103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.890300 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.890280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddxc\" (UniqueName: \"kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.890399 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.890336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.890904 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.890878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.892309 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.892291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.892466 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.892450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.906668 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.906637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddxc\" (UniqueName: \"kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc\") pod \"maas-api-5fcf74d56b-8qflx\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:52.991062 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.990997 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-maas-api-tls\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:52.991062 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.991051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-metrics-certs\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:52.991216 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.991125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m979\" (UniqueName: \"kubernetes.io/projected/e9084db0-afb3-4706-9c8f-42cacb4f921a-kube-api-access-6m979\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:52.991216 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.991155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9084db0-afb3-4706-9c8f-42cacb4f921a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:52.997950 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:52.997933 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:53.092429 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.091932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m979\" (UniqueName: \"kubernetes.io/projected/e9084db0-afb3-4706-9c8f-42cacb4f921a-kube-api-access-6m979\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.092602 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.092489 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9084db0-afb3-4706-9c8f-42cacb4f921a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.092602 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.092551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-maas-api-tls\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.092741 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.092669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-metrics-certs\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.093947 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.093915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9084db0-afb3-4706-9c8f-42cacb4f921a-openshift-service-ca\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.095438 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.095416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-maas-api-tls\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.095438 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.095423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9084db0-afb3-4706-9c8f-42cacb4f921a-metrics-certs\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.102653 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.102626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m979\" (UniqueName: \"kubernetes.io/projected/e9084db0-afb3-4706-9c8f-42cacb4f921a-kube-api-access-6m979\") pod \"maas-api-55d65dcfc4-chbk6\" (UID: \"e9084db0-afb3-4706-9c8f-42cacb4f921a\") " pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.137600 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.137570 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:53.280055 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.280028 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:08:53.282782 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:08:53.282749 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e16d699_e90d_41c6_a3a6_f37c1f9b5bc4.slice/crio-beb14d5f5255daff82604bca85a91d722a33e69579e6dc55e6cb7a7517ad68ab WatchSource:0}: Error finding container beb14d5f5255daff82604bca85a91d722a33e69579e6dc55e6cb7a7517ad68ab: Status 404 returned error can't find the container with id beb14d5f5255daff82604bca85a91d722a33e69579e6dc55e6cb7a7517ad68ab Sep 10 14:08:53.284457 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.284436 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:08:53.437192 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:53.437165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6"] Sep 10 14:08:53.438294 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:08:53.438264 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9084db0_afb3_4706_9c8f_42cacb4f921a.slice/crio-13995916645e40ddb028379302e54c6c788778839caca74e12f534716fa9c2ff WatchSource:0}: Error finding container 13995916645e40ddb028379302e54c6c788778839caca74e12f534716fa9c2ff: Status 404 returned error can't find the container with id 13995916645e40ddb028379302e54c6c788778839caca74e12f534716fa9c2ff Sep 10 14:08:54.013874 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:54.013841 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" event={"ID":"e9084db0-afb3-4706-9c8f-42cacb4f921a","Type":"ContainerStarted","Data":"13995916645e40ddb028379302e54c6c788778839caca74e12f534716fa9c2ff"} Sep 10 14:08:54.015038 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:54.015017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" event={"ID":"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4","Type":"ContainerStarted","Data":"beb14d5f5255daff82604bca85a91d722a33e69579e6dc55e6cb7a7517ad68ab"} Sep 10 14:08:55.021027 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:55.020997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" event={"ID":"e9084db0-afb3-4706-9c8f-42cacb4f921a","Type":"ContainerStarted","Data":"35292fa64f2bea2be9837efd450048434eeb5ac3363a3b1306e91481e639d92a"} Sep 10 14:08:55.021385 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:55.021061 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:08:55.051919 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:55.051869 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" podStartSLOduration=2.483439251 podStartE2EDuration="3.051850531s" totalImagesPullingTime="568.41128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:08:55.048888958 +0000 UTC m=+2310.658673870" watchObservedRunningTime="2026-09-10 14:08:55.051850531 +0000 UTC m=+2310.661635446" Sep 10 14:08:56.027736 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:56.027693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" event={"ID":"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4","Type":"ContainerStarted","Data":"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02"} Sep 10 14:08:56.028157 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:56.027998 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:08:56.060435 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:08:56.060386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" podStartSLOduration=2.356447837 podStartE2EDuration="4.060360323s" totalImagesPullingTime="1.703912486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:08:56.058448615 +0000 UTC m=+2311.668233527" watchObservedRunningTime="2026-09-10 14:08:56.060360323 +0000 UTC m=+2311.670145236" Sep 10 14:09:00.640829 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:00.640790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:09:01.033026 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.033001 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55d65dcfc4-chbk6" Sep 10 14:09:01.119959 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.119928 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:09:01.120164 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.120115 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" podUID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" containerName="maas-api" containerID="cri-o://314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02" gracePeriod=30 Sep 10 14:09:01.125113 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.125091 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:09:01.369969 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.369947 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:09:01.556182 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.556118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs\") pod \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " Sep 10 14:09:01.556182 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.556164 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls\") pod \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " Sep 10 14:09:01.556389 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.556198 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca\") pod \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " Sep 10 14:09:01.556389 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.556228 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fddxc\" (UniqueName: \"kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc\") pod \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\" (UID: \"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4\") " Sep 10 14:09:01.556632 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.556605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" (UID: "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:09:01.558269 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.558239 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" (UID: "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:09:01.558395 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.558316 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" (UID: "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:09:01.558395 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.558326 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc" (OuterVolumeSpecName: "kube-api-access-fddxc") pod "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" (UID: "1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4"). InnerVolumeSpecName "kube-api-access-fddxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:09:01.657101 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.657073 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddxc\" (UniqueName: \"kubernetes.io/projected/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-kube-api-access-fddxc\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:09:01.657101 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.657102 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-metrics-certs\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:09:01.657537 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.657117 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-maas-api-tls\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:09:01.657537 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:01.657130 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4-openshift-service-ca\") on node \"ip-10-0-133-208.ec2.internal\" DevicePath \"\"" Sep 10 14:09:02.063077 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.063045 2559 generic.go:356] "Generic (PLEG): container finished" podID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" containerID="314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02" exitCode=0 Sep 10 14:09:02.063263 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.063086 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" Sep 10 14:09:02.063263 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.063122 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" event={"ID":"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4","Type":"ContainerDied","Data":"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02"} Sep 10 14:09:02.063263 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.063177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx" event={"ID":"1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4","Type":"ContainerDied","Data":"beb14d5f5255daff82604bca85a91d722a33e69579e6dc55e6cb7a7517ad68ab"} Sep 10 14:09:02.063263 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.063200 2559 scope.go:117] "RemoveContainer" containerID="314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02" Sep 10 14:09:02.072772 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.072754 2559 scope.go:117] "RemoveContainer" containerID="314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02" Sep 10 14:09:02.073014 ip-10-0-133-208 kubenswrapper[2559]: E0910 14:09:02.072997 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02\": container with ID starting with 314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02 not found: ID does not exist" containerID="314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02" Sep 10 14:09:02.073059 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.073022 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02"} err="failed to get container status \"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02\": rpc error: code = NotFound desc = could not find container \"314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02\": container with ID starting with 314dba1046fea01f4fb73794f37ff37f673cfd3360363a941cd8181a71421f02 not found: ID does not exist" Sep 10 14:09:02.099745 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.099723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:09:02.107631 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.107608 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fcf74d56b-8qflx"] Sep 10 14:09:02.918543 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:02.918506 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" path="/var/lib/kubelet/pods/1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4/volumes" Sep 10 14:09:07.601957 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:07.601924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:09:13.493564 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:13.493529 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:09:20.283303 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:20.283223 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-zct4z"] Sep 10 14:09:29.227386 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:29.227334 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-88fwh_82925dd0-59f5-4df3-8c5c-6c0cb01126fb/manager/0.log" Sep 10 14:09:29.492842 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:29.492772 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-gqbms_7da1dc22-e1bd-4774-941c-04f3e4f85464/copy-manifests/0.log" Sep 10 14:09:29.500500 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:29.500471 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-gqbms_7da1dc22-e1bd-4774-941c-04f3e4f85464/manager/0.log" Sep 10 14:09:30.080637 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:30.080605 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55d65dcfc4-chbk6_e9084db0-afb3-4706-9c8f-42cacb4f921a/maas-api/0.log" Sep 10 14:09:30.460575 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:30.460501 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9qmff_975629be-325e-438b-9ce7-8ff3115c4497/postgres/0.log" Sep 10 14:09:31.405213 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:31.405181 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6dh2p_7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c/manager/0.log" Sep 10 14:09:31.521924 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:31.521898 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4pspg_3541f750-d0f7-453c-92a9-f9ffe38a8741/manager/0.log" Sep 10 14:09:31.892209 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:31.892184 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-zct4z_c494eafb-1357-4883-a003-50b78205641f/limitador/0.log" Sep 10 14:09:32.483696 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:32.483666 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nnn7m_0777e25d-1995-4f94-b993-b93c755b9b15/discovery/0.log" Sep 10 14:09:32.727184 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:32.727154 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66cb58b9b9-xjgxc_8f50a90b-6907-4185-9fe1-26661d8f2f86/kube-auth-proxy/0.log" Sep 10 14:09:32.854162 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:32.854135 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/1.log" Sep 10 14:09:33.100019 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:33.099997 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 14:09:33.222727 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:33.222666 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-676fd4f68-q9ztc_0dbe2dc5-3878-4994-be23-bcedf15db377/router/0.log" Sep 10 14:09:33.602810 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:33.602785 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-srkkm_f706a784-b20a-4cc0-af04-2c8f39b61a76/main/0.log" Sep 10 14:09:40.014859 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:40.014824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:09:40.110737 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:40.110705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:09:43.424756 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:43.424719 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kh6q9_7954047b-1d2d-4a68-8df2-db357abdf494/global-pull-secret-syncer/0.log" Sep 10 14:09:43.483031 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:43.483005 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" Sep 10 14:09:43.703203 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:43.703133 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-208.ec2.internal_4a5b2711db4d2239ddb842b9e3de4842/haproxy/0.log" Sep 10 14:09:48.759448 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:48.759416 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-6dh2p_7d5fd2e3-f1ad-4bb1-8840-d226f33e1b5c/manager/0.log" Sep 10 14:09:48.887723 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:48.887697 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-4pspg_3541f750-d0f7-453c-92a9-f9ffe38a8741/manager/0.log" Sep 10 14:09:49.207723 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:49.207643 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-zct4z_c494eafb-1357-4883-a003-50b78205641f/limitador/0.log" Sep 10 14:09:51.182996 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:51.182959 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-dvsk4_d16dc7a4-8ad9-4e42-91aa-a9ac7ab00ae9/cluster-monitoring-operator/0.log" Sep 10 14:09:51.409238 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:51.409213 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4q6cj_41ab9a90-3f87-4d36-bf7f-0de5c169e3f3/node-exporter/0.log" Sep 10 14:09:51.436356 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:51.436296 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4q6cj_41ab9a90-3f87-4d36-bf7f-0de5c169e3f3/kube-rbac-proxy/0.log" Sep 10 14:09:51.463745 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:51.463729 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4q6cj_41ab9a90-3f87-4d36-bf7f-0de5c169e3f3/init-textfile/0.log" Sep 10 14:09:52.214050 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:52.214003 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-fst68_8c521425-b190-49ee-bc34-3c885fce1786/prometheus-operator-admission-webhook/0.log" Sep 10 14:09:53.682685 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:53.682652 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-49wvj_b28758f1-b930-4ef4-8371-b2cb8c6e4ef4/networking-console-plugin/0.log" Sep 10 14:09:54.321597 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.321566 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/1.log" Sep 10 14:09:54.326254 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.326231 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgjcm_6aeb68d5-6782-45d0-8aff-fa15cd8e9266/console-operator/2.log" Sep 10 14:09:54.655943 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.655868 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h"] Sep 10 14:09:54.656255 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.656242 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" containerName="maas-api" Sep 10 14:09:54.656300 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.656257 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" containerName="maas-api" Sep 10 14:09:54.656361 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.656351 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e16d699-e90d-41c6-a3a6-f37c1f9b5bc4" containerName="maas-api" Sep 10 14:09:54.659597 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.659580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.662387 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.662338 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mbn6s\"/\"kube-root-ca.crt\"" Sep 10 14:09:54.663311 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.663290 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mbn6s\"/\"openshift-service-ca.crt\"" Sep 10 14:09:54.663450 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.663318 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mbn6s\"/\"default-dockercfg-2lwpv\"" Sep 10 14:09:54.670991 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.670969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h"] Sep 10 14:09:54.778454 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.778428 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-podres\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.778751 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.778474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-proc\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.778751 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.778532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbfr\" (UniqueName: \"kubernetes.io/projected/94c34648-4f98-47ee-ac92-e35764d1ed34-kube-api-access-fpbfr\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.778751 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.778577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-lib-modules\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.778751 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.778612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-sys\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879249 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-podres\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879336 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-proc\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879336 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbfr\" (UniqueName: \"kubernetes.io/projected/94c34648-4f98-47ee-ac92-e35764d1ed34-kube-api-access-fpbfr\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879336 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-lib-modules\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879336 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-sys\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879517 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-proc\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879517 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-podres\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879517 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-sys\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.879517 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.879469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94c34648-4f98-47ee-ac92-e35764d1ed34-lib-modules\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.888901 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.888882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbfr\" (UniqueName: \"kubernetes.io/projected/94c34648-4f98-47ee-ac92-e35764d1ed34-kube-api-access-fpbfr\") pod \"perf-node-gather-daemonset-q7w6h\" (UID: \"94c34648-4f98-47ee-ac92-e35764d1ed34\") " pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.943269 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.943213 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84f7c4c7dd-vm7h4_0d4637f1-7a0b-427b-b354-03aaa39ca196/console/0.log" Sep 10 14:09:54.971495 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.971477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:54.982271 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:54.982252 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-wtptv_669c481b-fdf8-45f2-b199-a34dfd1dcc52/download-server/0.log" Sep 10 14:09:55.245515 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.245492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h"] Sep 10 14:09:55.247109 ip-10-0-133-208 kubenswrapper[2559]: W0910 14:09:55.247081 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94c34648_4f98_47ee_ac92_e35764d1ed34.slice/crio-01bf90e536905912c93eb616a368a0b70285851134d25ff387e3a3a41ffbb6de WatchSource:0}: Error finding container 01bf90e536905912c93eb616a368a0b70285851134d25ff387e3a3a41ffbb6de: Status 404 returned error can't find the container with id 01bf90e536905912c93eb616a368a0b70285851134d25ff387e3a3a41ffbb6de Sep 10 14:09:55.375607 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.368987 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" event={"ID":"94c34648-4f98-47ee-ac92-e35764d1ed34","Type":"ContainerStarted","Data":"ae9d53511aa5d70da761a2a7d878d5c083866a0ecf0fef1905e5da5e110822b9"} Sep 10 14:09:55.375607 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.369147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" event={"ID":"94c34648-4f98-47ee-ac92-e35764d1ed34","Type":"ContainerStarted","Data":"01bf90e536905912c93eb616a368a0b70285851134d25ff387e3a3a41ffbb6de"} Sep 10 14:09:55.375607 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.369225 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:09:55.387734 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.387690 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" podStartSLOduration=1.387677009 podStartE2EDuration="1.387677009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:09:55.386709291 +0000 UTC m=+2370.996494205" watchObservedRunningTime="2026-09-10 14:09:55.387677009 +0000 UTC m=+2370.997461922" Sep 10 14:09:55.636333 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:55.636304 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-l5l4s_97c642c5-28bd-4ea9-97dd-311e2d943a9a/volume-data-source-validator/0.log" Sep 10 14:09:56.888238 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:56.888208 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccxnk_9936248a-cfed-420b-914f-b5ebc63997ea/dns/0.log" Sep 10 14:09:56.944651 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:56.944615 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ccxnk_9936248a-cfed-420b-914f-b5ebc63997ea/kube-rbac-proxy/0.log" Sep 10 14:09:57.261193 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:57.261164 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wzz2s_5828c67b-c93e-4af9-946e-2da852db81e3/dns-node-resolver/0.log" Sep 10 14:09:57.953938 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:57.953911 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxlmv_80cbebcd-aa00-4d3a-98ec-0e86bb4275e7/node-ca/0.log" Sep 10 14:09:59.438772 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.438741 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nnn7m_0777e25d-1995-4f94-b993-b93c755b9b15/discovery/0.log" Sep 10 14:09:59.496089 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.496066 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66cb58b9b9-xjgxc_8f50a90b-6907-4185-9fe1-26661d8f2f86/kube-auth-proxy/0.log" Sep 10 14:09:59.535203 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.535174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/0.log" Sep 10 14:09:59.543208 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.543191 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/1.log" Sep 10 14:09:59.620112 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.620083 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 14:09:59.647481 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:09:59.647434 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-676fd4f68-q9ztc_0dbe2dc5-3878-4994-be23-bcedf15db377/router/0.log" Sep 10 14:10:00.689917 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:00.689885 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n9twt_d373a59e-ec7a-4648-b392-e750f5f3d554/serve-healthcheck-canary/0.log" Sep 10 14:10:01.331838 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.331809 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dkbfp_2fd2d410-6c72-4e23-8d17-d1554cbe69e8/insights-operator/0.log" Sep 10 14:10:01.332483 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.332465 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-dkbfp_2fd2d410-6c72-4e23-8d17-d1554cbe69e8/insights-operator/1.log" Sep 10 14:10:01.384536 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.384505 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mbn6s/perf-node-gather-daemonset-q7w6h" Sep 10 14:10:01.597464 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.597362 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9wl8_213d309e-3e5d-4df8-9ea1-a3f9010e85ab/kube-rbac-proxy/0.log" Sep 10 14:10:01.664507 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.664485 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9wl8_213d309e-3e5d-4df8-9ea1-a3f9010e85ab/exporter/0.log" Sep 10 14:10:01.725055 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:01.725018 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v9wl8_213d309e-3e5d-4df8-9ea1-a3f9010e85ab/extractor/0.log" Sep 10 14:10:04.167082 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.167038 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nnn7m_0777e25d-1995-4f94-b993-b93c755b9b15/discovery/0.log" Sep 10 14:10:04.303800 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.303774 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66cb58b9b9-xjgxc_8f50a90b-6907-4185-9fe1-26661d8f2f86/kube-auth-proxy/0.log" Sep 10 14:10:04.355528 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.355504 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/0.log" Sep 10 14:10:04.363497 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.363480 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/1.log" Sep 10 14:10:04.437718 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.437694 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 14:10:04.474514 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.474497 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-676fd4f68-q9ztc_0dbe2dc5-3878-4994-be23-bcedf15db377/router/0.log" Sep 10 14:10:04.942192 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:04.942125 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-88fwh_82925dd0-59f5-4df3-8c5c-6c0cb01126fb/manager/0.log" Sep 10 14:10:05.110282 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:05.110254 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-gqbms_7da1dc22-e1bd-4774-941c-04f3e4f85464/manager/0.log" Sep 10 14:10:05.223688 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:05.223626 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-94d598c87-gqbms_7da1dc22-e1bd-4774-941c-04f3e4f85464/copy-manifests/0.log" Sep 10 14:10:06.431302 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.431235 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nnn7m_0777e25d-1995-4f94-b993-b93c755b9b15/discovery/0.log" Sep 10 14:10:06.478236 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.478210 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66cb58b9b9-xjgxc_8f50a90b-6907-4185-9fe1-26661d8f2f86/kube-auth-proxy/0.log" Sep 10 14:10:06.500718 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.500694 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/0.log" Sep 10 14:10:06.508928 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.508909 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-fnzdk_cf9f0176-0f4d-447a-a297-2c03237e1f9b/istio-proxy/1.log" Sep 10 14:10:06.565358 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.565329 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7wzmf_844b6c0b-9f1b-4270-b728-adad14093bec/payload-processing/0.log" Sep 10 14:10:06.580866 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.580847 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-676fd4f68-q9ztc_0dbe2dc5-3878-4994-be23-bcedf15db377/router/0.log" Sep 10 14:10:06.766648 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.766627 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qb8vk_5f051725-362f-4eb1-b1b7-38facdfcfef2/openshift-lws-operator/0.log" Sep 10 14:10:06.771830 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:06.771808 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx2sk_824f69e2-4b2e-44c9-a072-aa093b3f7b92/konnectivity-agent/0.log" limit=1024 Sep 10 14:10:12.091114 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:12.091088 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-z2bk8_ff92bed8-0c9a-4969-bbe8-9e322591cbc2/migrator/0.log" Sep 10 14:10:12.122240 ip-10-0-133-208 kubenswrapper[2559]: I0910 14:10:12.122215 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-z2bk8_ff92bed8-0c9a-4969-bbe8-9e322591cbc2/graceful-termination/0.log"